{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,29]],"date-time":"2026-05-29T14:54:40Z","timestamp":1780066480422,"version":"3.54.0"},"reference-count":70,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2020,1,20]],"date-time":"2020-01-20T00:00:00Z","timestamp":1579478400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,1,20]],"date-time":"2020-01-20T00:00:00Z","timestamp":1579478400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2020,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Assessing the risks of software vulnerabilities is a key process of software development and security management. This assessment requires to consider multiple factors (technical features, operational environment, involved assets, status of the vulnerability lifecycle, etc.) and may depend from the assessor\u2019s knowledge and skills. In this work, we tackle with an important part of this problem by measuring the accuracy of<jats:italic>technical<\/jats:italic>vulnerability assessments by assessors with different level and type of knowledge. We report an experiment to compare how accurately students with different technical education and security professionals are able to assess the severity of software vulnerabilities with the Common Vulnerability Scoring System (v3) industry methodology. Our results could be useful for increasing awareness about the intrinsic subtleties of vulnerability risk assessment and possibly better compliance with regulations. With respect to academic education, professional training and human resources selections our work suggests that measuring the effects of knowledge and expertise on the accuracy of software security assessments is feasible albeit not easy.<\/jats:p>","DOI":"10.1007\/s10664-019-09797-4","type":"journal-article","created":{"date-parts":[[2020,1,20]],"date-time":"2020-01-20T05:16:18Z","timestamp":1579497378000},"page":"1063-1094","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":36,"title":["Measuring the accuracy of software vulnerability assessments: experiments with students and professionals"],"prefix":"10.1007","volume":"25","author":[{"given":"Luca","family":"Allodi","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marco","family":"Cremonini","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fabio","family":"Massacci","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Woohyun","family":"Shim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2020,1,20]]},"reference":[{"key":"9797_CR1","doi-asserted-by":"crossref","unstructured":"Acar Y, Backes M, Fahl S, Kim D, Mazurek ML, Stransky C (2016) You Get where you\u2019re looking for: The impact of information sources on code security. In: Proceedings of the IEEE symposium on security and privacy (SP). IEEE, pp 289\u2013305","DOI":"10.1109\/SP.2016.25"},{"key":"9797_CR2","doi-asserted-by":"crossref","unstructured":"Acar Y, Backes M, Fahl S, Garfinkel S, Kim D, Mazurek ML, Stransky C (2017) Comparing the usability of cryptographic APIs. In: Proceedings of the IEEE symposium on security and privacy (SP). IEEE, pp 154\u2013171","DOI":"10.1109\/SP.2017.52"},{"key":"9797_CR3","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1007\/978-3-642-04898-2_161","volume-title":"International Encyclopedia of Statistical Science","author":"Alan Agresti","year":"2011","unstructured":"Agresti A, Kateri M (2011) Categorical data analysis. In: Lovric M (ed) International encyclopedia of statistical science. Springer, Berlin, pp 206\u2013208"},{"issue":"1","key":"9797_CR4","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2630069","volume":"17","author":"Luca Allodi","year":"2014","unstructured":"Allodi L, Massacci F (2014) Comparing vulnerability severity and exploits using case-control studies. ACM Transactions on Information and System Security (TISSEC) 17(1)","journal-title":"ACM Transactions on Information and System Security"},{"issue":"8","key":"9797_CR5","doi-asserted-by":"publisher","first-page":"1606","DOI":"10.1111\/risa.12864","volume":"37","author":"L Allodi","year":"2017","unstructured":"Allodi L, Massacci F (2017) Security events and vulnerability data for cybersecurity risk estimation. Risk Anal. 37(8):1606\u20131627","journal-title":"Risk Anal."},{"key":"9797_CR6","doi-asserted-by":"crossref","unstructured":"Allodi L, Biagioni S, Crispo B, Labunets K, Massacci F, Santos W (2017) Estimating the assessment difficulty of CVSS environmental metrics: an experiment. In: Proceedings of the international conference on future data and security engineering. Springer, pp 23\u201339","DOI":"10.1007\/978-3-319-70004-5_2"},{"issue":"1","key":"9797_CR7","doi-asserted-by":"publisher","first-page":"84","DOI":"10.1109\/MSP.2005.23","volume":"3","author":"B Arkin","year":"2005","unstructured":"Arkin B, Stender S, McGraw G (2005) Software penetration testing. IEEE Security & Privacy 3(1):84\u201387","journal-title":"IEEE Security & Privacy"},{"key":"9797_CR8","first-page":"17","volume-title":"Assessment and Teaching of 21st Century Skills","author":"Marilyn Binkley","year":"2011","unstructured":"Binkley M, Erstad O, Herman J, Raizen S, Ripley M, Miller-Ricci M, Rumble M (2012) Defining twenty-first century skills. In: Griffin P, McGaw B, Care E (eds) Assessment and teaching of 21st century skills. Springer, Dordrecht, pp 17\u201366"},{"key":"9797_CR9","doi-asserted-by":"crossref","unstructured":"Bozorgi M, Saul LK, Savage S, Voelker GM (2010) Beyond heuristics: Learning to classify vulnerabilities and predict exploits. In: Proceedings of the 16th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, pp 105\u2013114","DOI":"10.1145\/1835804.1835821"},{"issue":"2","key":"9797_CR10","doi-asserted-by":"publisher","first-page":"1153","DOI":"10.1109\/COMST.2015.2494502","volume":"18","author":"AL Buczak","year":"2016","unstructured":"Buczak AL, Guven E (2016) A survey of data mining and machine learning methods for cyber security intrusion detection. IEEE Communications Surveys & Tutorials 18(2):1153\u20131176","journal-title":"IEEE Communications Surveys & Tutorials"},{"issue":"3","key":"9797_CR11","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1109\/MC.2018.2883567","volume":"52","author":"DL Burley","year":"2019","unstructured":"Burley DL, Lewis AH Jr (2019) Cybersecurity curricula 2017 and boeing: Linking curricular guidance to professional practice. Computer 52(3):29\u201337","journal-title":"Computer"},{"issue":"2","key":"9797_CR12","doi-asserted-by":"publisher","first-page":"24","DOI":"10.1145\/2556936","volume":"57","author":"DL Burley","year":"2014","unstructured":"Burley DL, Eisenberg J, Goodman SE (2014) Would cybersecurity professionalization help address the cybersecurity crisis? Commun. ACM 57(2):24\u201327","journal-title":"Commun. ACM"},{"key":"9797_CR13","unstructured":"Camerer CF, Johnson EJ (1991) The process-performance paradox in expert judgment: How can experts know so much and predict so badly?. In: Ericsson KA, Smith J (eds) Toward a general theory of expertise: Prospects and limits. Cambridge University Press, pp 195\u2013217"},{"key":"9797_CR14","doi-asserted-by":"crossref","unstructured":"Colesky M, Hoepman JH, Hillen C (2016) A critical analysis of privacy design strategies. In: Proceedings of the IEEE security and privacy workshops (SPW). IEEE, pp 33\u201340","DOI":"10.1109\/SPW.2016.23"},{"key":"9797_CR15","doi-asserted-by":"crossref","unstructured":"Conklin W, Bishop M, et al. (2018) Contrasting the csec 2017 and the cae designation requirements. In: Proceedings of the 51st Hawaii international conference on system sciences","DOI":"10.24251\/HICSS.2018.306"},{"key":"9797_CR16","doi-asserted-by":"crossref","unstructured":"Conklin WA, Cline RE, Roosa T (2014) Re-engineering cybersecurity education in the US: an analysis of the critical factors. In: Proceedings of the 47th Hawaii international conference on system sciences (HICSS). IEEE, pp 2006\u20132014","DOI":"10.1109\/HICSS.2014.254"},{"key":"9797_CR17","doi-asserted-by":"crossref","unstructured":"Conti M, Dargahi T, Dehghantanha A (2018) Cyber threat intelligence: challenges and opportunities. Advances in Information Security, 70, Springer International Publishing","DOI":"10.1007\/978-3-319-73951-9"},{"issue":"4","key":"9797_CR18","doi-asserted-by":"publisher","first-page":"747","DOI":"10.1007\/s00355-017-1034-z","volume":"48","author":"F Dietrich","year":"2017","unstructured":"Dietrich F, List C (2017) Probabilistic opinion pooling generalized. Part one: general agendas. Soc. Choice Welf. 48(4):747\u2013786","journal-title":"Soc. Choice Welf."},{"key":"9797_CR19","doi-asserted-by":"crossref","unstructured":"Doynikova E, Kotenko I (2017) CVSS-based probabilistic risk assessment for cyber situational awareness and countermeasure selection. In: Proceedings of the 25th Euromicro international conference on parallel, distributed and network-based processing (PDP). IEEE, pp 346\u2013353","DOI":"10.1109\/PDP.2017.44"},{"key":"9797_CR20","doi-asserted-by":"crossref","unstructured":"Edmundson A, Holtkamp B, Rivera E, Finifter M, Mettler A, Wagner D (2013) An empirical study on the effectiveness of security code review. In: Proceedings of the international symposium on engineering secure software and systems. Springer, pp 197\u2013212","DOI":"10.1007\/978-3-642-36563-8_14"},{"key":"9797_CR21","unstructured":"ENISA (2017) Priorities for EU research - analysis of the ECSO Strategic Research and Innovation Agenda (SRIA). https:\/\/www.enisa.europa.eu\/publications\/priorities-for-eu-research"},{"key":"9797_CR22","unstructured":"FIRST (2015) Common vulnerability scoring system v3.0: Specification Document. Tech. rep., FIRST. http:\/\/www.first.org\/cvss"},{"issue":"2","key":"9797_CR23","first-page":"50","volume":"40","author":"D Geer","year":"2015","unstructured":"Geer D (2015) For good measure: The undiscovered. login:: the magazine of USENIX & SAGE 40(2):50\u201352","journal-title":"login:: the magazine of USENIX & SAGE"},{"key":"9797_CR24","unstructured":"Hallett J, Larson R, Rashid A (2018) Mirror, mirror, on the wall: What are we teaching them all? Characterising the focus of cybersecurity curricular frameworks. In: Proceedings of the USENIX workshop on advances in security education (ASE 18), USENIX Association, Baltimore, MD"},{"key":"9797_CR25","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1016\/j.cose.2015.04.012","volume":"53","author":"H Holm","year":"2015","unstructured":"Holm H, Afridi KK (2015) An expert-based investigation of the common vulnerability scoring system. Computers & Security 53:18\u201330","journal-title":"Computers & Security"},{"issue":"3","key":"9797_CR26","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1109\/MC.2018.2883334","volume":"52","author":"M Hudnall","year":"2019","unstructured":"Hudnall M (2019) Educational and workforce cybersecurity frameworks: comparing, contrasting, and mapping. Computer 52(3):18\u201328","journal-title":"Computer"},{"issue":"3","key":"9797_CR27","doi-asserted-by":"publisher","first-page":"369","DOI":"10.1007\/s10270-010-0154-z","volume":"10","author":"S Islam","year":"2011","unstructured":"Islam S, Mouratidis H, J\u00fcrjens J (2011) A framework to support alignment of secure software engineering with legal regulations. Software & Systems Modeling 10 (3):369\u2013394","journal-title":"Software & Systems Modeling"},{"key":"9797_CR28","unstructured":"ISO (2008) ISO\/IEC 27005 Information technology \u2013 Security techniques \u2013 Information security risk management. Tech. rep., http:\/\/www.iso.org\/iso\/catalogue_detail?csnumber=56742"},{"key":"9797_CR29","doi-asserted-by":"crossref","unstructured":"Jacobs J, Romanosky S, Adjerid I, Baker W (2019) Improving vulnerability remediation through better exploit prediction. In: Proceedings of the workshop on the economics of information security. https:\/\/weis2019.econinfosec.org\/wp-content\/uploads\/sites\/6\/2019\/05\/WEIS_2019_paper_53.pdf","DOI":"10.1093\/cybsec\/tyaa015"},{"key":"9797_CR30","doi-asserted-by":"crossref","unstructured":"Joint Task Force on Cybersecurity Education (2017) Curriculum guidelines for post-secondary degree programs in cybersecurity (CSEC2017). https:\/\/www.acm.org\/binaries\/content\/assets\/education\/curricula-recommendations\/csec2017.pdf","DOI":"10.1145\/3422808"},{"issue":"1","key":"9797_CR31","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1207\/S15326985EP3801_4","volume":"38","author":"S Kalyuga","year":"2003","unstructured":"Kalyuga S, Ayres P, Chandler P, Sweller J (2003) The expertise reversal effect. Educational Psychologist 38(1):23\u201331","journal-title":"Educational Psychologist"},{"key":"9797_CR32","doi-asserted-by":"crossref","unstructured":"Katsantonis M, Fouliras P, Mavridis I (2017) Conceptual analysis of cyber security education based on live competitions. In: Proceedings of Global Engineering Education Conference (EDUCON). IEEE, pp 771\u2013779","DOI":"10.1109\/EDUCON.2017.7942934"},{"key":"9797_CR33","doi-asserted-by":"publisher","first-page":"111","DOI":"10.1007\/978-3-319-95831-6_9","volume-title":"Cognitive Biases in Visualizations","author":"Donald R. Kretz","year":"2018","unstructured":"Kretz DR (2018) Experimentally evaluating bias-reducing visual analytics techniques in intelligence analysis. In: Geoffrey E (ed) Cognitive biases in visualizations. Springer, Cham, pp 111\u2013135"},{"issue":"8","key":"9797_CR34","doi-asserted-by":"publisher","first-page":"2184","DOI":"10.1016\/j.tele.2018.08.006","volume":"35","author":"E van Laar","year":"2018","unstructured":"van Laar E, van Deursen AJ, van Dijk JA, de Haan J (2018) 21st-century digital skills instrument aimed at working professionals: Conceptual development and empirical validation. Telematics and Informatics 35(8):2184\u20132200","journal-title":"Telematics and Informatics"},{"issue":"6","key":"9797_CR35","doi-asserted-by":"publisher","first-page":"3017","DOI":"10.1007\/s10664-017-9502-8","volume":"22","author":"K Labunets","year":"2017","unstructured":"Labunets K, Massacci F, Paci F, Marczak S, de Oliveira FM (2017) Model comprehension for security risk assessment: an empirical comparison of tabular vs. graphical representations. Empir. Softw. Eng. 22(6):3017\u20133056","journal-title":"Empir. Softw. Eng."},{"key":"9797_CR36","doi-asserted-by":"crossref","unstructured":"Lichtenstein S, Fischhoff B, Phillips LD (1982) Calibration of probabilities: The state of the art to 1980. In: Kahneman D, Slovic P, Tversky A (eds) Judgment under uncertainty: heuristics and biases. Cambridge University Press, pp 306\u2013334","DOI":"10.1017\/CBO9780511809477.023"},{"key":"9797_CR37","unstructured":"Marks J (2018) NIST teams up with IBM Watson to rate how dangerous computer bugs are. https:\/\/www.nextgov.com\/cybersecurity\/2018\/11\/nist-teams-ibms-watson-rate-how-dangerous-computer-bugs-are\/152545\/"},{"issue":"6","key":"9797_CR38","doi-asserted-by":"publisher","first-page":"66","DOI":"10.1109\/MSP.2013.155","volume":"11","author":"A McGettrick","year":"2013","unstructured":"McGettrick A (2013) Toward effective cybersecurity education. IEEE Security & Privacy 11(6):66\u201368","journal-title":"IEEE Security & Privacy"},{"key":"9797_CR39","doi-asserted-by":"crossref","unstructured":"McGraw G (2006) Software security: building security in, vol 1. Addison-Wesley Professional","DOI":"10.1109\/ISSRE.2006.43"},{"key":"9797_CR40","unstructured":"Mell P, Scarfone K, Romanosky S (2007) A complete guide to the common vulnerability scoring system version 2.0. Tech. rep., FIRST, Available at http:\/\/www.first.org\/cvss"},{"issue":"2","key":"9797_CR41","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1080\/07350015.1995.10524589","volume":"13","author":"BD Meyer","year":"1995","unstructured":"Meyer BD (1995) Natural and quasi-experiments in economics. Journal of Business & Economic Statistics 13(2):151\u2013161","journal-title":"Journal of Business & Economic Statistics"},{"key":"9797_CR42","unstructured":"Microsoft (2019) Microsoft security development lifecycle (SDL). https:\/\/www.microsoft.com\/en-us\/securityengineering\/sdl\/"},{"key":"9797_CR43","doi-asserted-by":"crossref","unstructured":"Morel B (2011) Artificial intelligence and the future of cybersecurity. In: Proceedings of the 4th ACM workshop on security and artificial intelligence. ACM, pp 93\u201398","DOI":"10.1145\/2046684.2046699"},{"key":"9797_CR44","doi-asserted-by":"crossref","unstructured":"Morrison P, Smith BH, Williams L (2017) Surveying security practice adherence in software development. In: Proceedings of Hot Topics in Science of Security: Symposium and Bootcamp. ACM, pp 85\u201394","DOI":"10.1145\/3055305.3055312"},{"key":"9797_CR45","doi-asserted-by":"publisher","first-page":"146","DOI":"10.1016\/j.infsof.2018.05.011","volume":"102","author":"P Morrison","year":"2018","unstructured":"Morrison P, Moye D, Pandita R, Williams L (2018) Mapping the field of software life cycle security metrics. Inf. Softw. Technol. 102:146\u2013159","journal-title":"Inf. Softw. Technol."},{"issue":"450","key":"9797_CR46","doi-asserted-by":"publisher","first-page":"449","DOI":"10.1080\/01621459.2000.10474219","volume":"95","author":"SA Murphy","year":"2000","unstructured":"Murphy SA, Van der Vaart AW (2000) On profile likelihood. J. Am. Stat. Assoc. 95(450):449\u2013465","journal-title":"J. Am. Stat. Assoc."},{"issue":"2","key":"9797_CR47","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1111\/j.2041-210x.2012.00261.x","volume":"4","author":"S Nakagawa","year":"2013","unstructured":"Nakagawa S, Schielzeth H (2013) A general and simple method for obtaining r2 from generalized linear mixed-effects models. Methods Ecol. Evol. 4(2):133\u2013142","journal-title":"Methods Ecol. Evol."},{"key":"9797_CR48","unstructured":"NIST (2018) Vulnerability Description Ontology (VDO): a framework for characterizing vulnerabilities. https:\/\/csrc.nist.gov\/publications\/detail\/nistir\/8138\/draft"},{"key":"9797_CR49","unstructured":"Onarlioglu K, Yilmaz UO, Kirda E, Balzarotti D (2012) Insights into user behavior in dealing with internet attacks. In: Proceedings of the network and distributed system security symposium (NDSS), San Diego, CA"},{"key":"9797_CR50","unstructured":"OWASP (2019) OWASP risk rating methodology. https:\/\/www.owasp.org\/index.php\/OWASP_Risk_Rating_Methodology"},{"key":"9797_CR51","unstructured":"PCI-DSS (2018) Payment Card Industry (PCI) data security standard - requirements and security assessment procedures version 3.2.1. Tech. rep., https:\/\/www.pcisecuritystandards.org\/documents\/PCI_DSS_v3-2-1.pdf"},{"key":"9797_CR52","doi-asserted-by":"publisher","first-page":"182","DOI":"10.1016\/j.cose.2014.10.007","volume":"48","author":"R Reece","year":"2015","unstructured":"Reece R, Stahl BC (2015) The professionalisation of information security: Perspectives of UK practitioners. Computers & Security 48:182\u2013195","journal-title":"Computers & Security"},{"key":"9797_CR53","unstructured":"SafeCODE (2018) Fundamental practices for secure software development, third edition. https:\/\/safecode.org\/publications\/#safecodepublications-2362"},{"key":"9797_CR54","doi-asserted-by":"crossref","unstructured":"Salman I, Misirli AT, Juristo N (2015) Are students representatives of professionals in software engineering experiments?. In: Proceedings of the 37th international conference on software engineering (ICSE), vol 1, pp 666\u2013676","DOI":"10.1109\/ICSE.2015.82"},{"key":"9797_CR55","doi-asserted-by":"crossref","unstructured":"Santos H, Pereira T, Mendes I (2017) Challenges and reflections in designing cyber security curriculum. In: Proceedings of the world engineering education conference (EDUNINE). IEEE, pp 47\u201351","DOI":"10.1109\/EDUNINE.2017.7918179"},{"key":"9797_CR56","doi-asserted-by":"crossref","unstructured":"Scarfone K, Mell P (2009) An analysis of CVSS version 2 vulnerability scoring. In: Proceedings of the empirical software engineering and measurement (ESEM) conference, pp 516\u2013525","DOI":"10.1109\/ESEM.2009.5314220"},{"key":"9797_CR57","doi-asserted-by":"crossref","unstructured":"Shumba R, Ferguson-Boucher K, Sweedyk E, Taylor C, Franklin G, Turner C, Sande C, Acholonu G, Bace R, Hall L (2013) Cybersecurity, women and minorities: findings and recommendations from a preliminary investigation. In: Proceedings of the ITiCSE working group reports conference on Innovation and technology in computer science education-working group reports. ACM, pp 1\u201314","DOI":"10.1145\/2543882.2543883"},{"issue":"11","key":"9797_CR58","doi-asserted-by":"publisher","first-page":"1103","DOI":"10.1119\/1.1512659","volume":"70","author":"C Singh","year":"2002","unstructured":"Singh C (2002) When physical intuition fails. Am. J. Phys. 70(11):1103\u20131109","journal-title":"Am. J. Phys."},{"key":"9797_CR59","doi-asserted-by":"crossref","unstructured":"Sj\u00f8berg D, Anda B, Arisholm E, Dyb\u00e5 T, J\u00f8rgensen M, Karahasanovi\u0107 A, Vok\u00e1\u010d M (2003) Challenges and recommendations when increasing the realism of controlled software engineering experiments. In: Empirical methods and studies in software engineering, LNCS, vol 2765. Springer, Berlin, pp 24\u201338","DOI":"10.1007\/978-3-540-45143-3_3"},{"key":"9797_CR60","unstructured":"Spring J, Hatleback E, Householder AD, Manion A, Shick D (2018) White paper: Towards improving CVSS. Tech. rep., Carnegie Mellon University, Software Engineering Institute. https:\/\/resources.sei.cmu.edu\/library\/asset-view.cfm?assetID=538368"},{"key":"9797_CR61","unstructured":"The Parliament and the Council of European Union (2016a) Directive (EU) 2016\/1148. https:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/?uri=uriserv:OJ.L_.2016.194.01.0001.01.ENG&toc=OJ:L:2016:194:TOC"},{"key":"9797_CR62","unstructured":"The Parliament and the Council of European Union (2016b) Regulation (EU) 2016\/679. https:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/?qid=1532348683434&uri=CELEX:02016R0679-20160504"},{"key":"9797_CR63","unstructured":"Tripwire (2019) Advanced vulnerability risk scoring and prioritization. https:\/\/www.tripwire.com\/solutions\/vulnerability-and-risk-management\/vulnerability-risk-score-register\/"},{"key":"9797_CR64","doi-asserted-by":"publisher","first-page":"577","DOI":"10.1016\/j.chb.2017.03.010","volume":"72","author":"E Van Laar","year":"2017","unstructured":"Van Laar E, van Deursen AJ, van Dijk JA, de Haan J (2017) The relation between 21st-century skills and digital skills: a systematic literature review. Computers in Human Behavior 72:577\u2013588","journal-title":"Computers in Human Behavior"},{"key":"9797_CR65","volume-title":"Building secure software: How to avoid security problems the right way, portable documents","author":"J Viega","year":"2001","unstructured":"Viega J, McGraw GR (2001) Building secure software: How to avoid security problems the right way, portable documents. Pearson Education, London"},{"issue":"2","key":"9797_CR66","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1016\/j.cose.2005.02.002","volume":"24","author":"B Von Solms","year":"2005","unstructured":"Von Solms B (2005) Information security governance: COBIT or ISO 17799 or both? Computers & Security 24(2):99\u2013104","journal-title":"Computers & Security"},{"key":"9797_CR67","unstructured":"Wermke D, Mazurek M (2017) Security developer studies with GitHub users: Exploring a convenience sample. In: Proceedings of the symposium on usable privacy and security (SOUPS), USENIX Association, pp 81\u201395"},{"key":"9797_CR68","unstructured":"Williams BR, Chuvakin A (2012) PCI Compliance: Understand and implement effective PCI data security standard compliance. Syngress Elsevier"},{"key":"9797_CR69","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-29044-2","volume-title":"Experimentation in software engineering","author":"C Wohlin","year":"2012","unstructured":"Wohlin C, Runeson P, H\u00f6st M, Ohlsson MC, Regnell B, Wessl\u00e9n A (2012) Experimentation in software engineering, 1st edn. Springer, Berlin","edition":"1st edn."},{"issue":"4","key":"9797_CR70","first-page":"662","volume":"59","author":"M Workman","year":"2008","unstructured":"Workman M (2008) Wisecrackers: a theory-grounded investigation of phishing and pretext social engineering threats to information security. Journal of the Association for Information Science and Technology 59(4):662\u2013674","journal-title":"Journal of the Association for Information Science and Technology"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-019-09797-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s10664-019-09797-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-019-09797-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,30]],"date-time":"2024-07-30T10:02:56Z","timestamp":1722333776000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s10664-019-09797-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,1,20]]},"references-count":70,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,3]]}},"alternative-id":["9797"],"URL":"https:\/\/doi.org\/10.1007\/s10664-019-09797-4","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,1,20]]},"assertion":[{"value":"20 January 2020","order":1,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}