{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,4]],"date-time":"2026-04-04T01:33:03Z","timestamp":1775266383762,"version":"3.50.1"},"reference-count":103,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2022,8,6]],"date-time":"2022-08-06T00:00:00Z","timestamp":1659744000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.springer.com\/tdm"},{"start":{"date-parts":[[2022,8,6]],"date-time":"2022-08-06T00:00:00Z","timestamp":1659744000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.springer.com\/tdm"}],"funder":[{"DOI":"10.13039\/100000143","name":"division of computing and communication foundations","doi-asserted-by":"publisher","award":["1909516"],"award-info":[{"award-number":["1909516"]}],"id":[{"id":"10.13039\/100000143","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2022,11]]},"DOI":"10.1007\/s10664-022-10179-6","type":"journal-article","created":{"date-parts":[[2022,8,6]],"date-time":"2022-08-06T08:02:37Z","timestamp":1659772957000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":22,"title":["Do I really need all this work to find vulnerabilities?"],"prefix":"10.1007","volume":"27","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5881-4619","authenticated-orcid":false,"given":"Sarah","family":"Elder","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nusrat","family":"Zahan","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rui","family":"Shu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Monica","family":"Metro","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Valeri","family":"Kozarev","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tim","family":"Menzies","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Laurie","family":"Williams","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,8,6]]},"reference":[{"key":"10179_CR1","unstructured":"Ackerman E (2019) Upgrade to superhuman reflexes without feeling like a robot. IEEE Spectr. https:\/\/spectrum.ieee.org\/enabling-superhuman-reflexes-without-feeling-like-a-robot"},{"key":"10179_CR2","unstructured":"Alomar N, Wijesekera P, Qiu E, Egelman S (2020) \u201cyou\u2019ve got your nice list of bugs, now what?\u201d vulnerability discovery and management processes in the wild. In: Sixteenth Symposium on Usable Privacy and Security ({SOUPS} 2020), pp 319\u2013339"},{"issue":"9","key":"10179_CR3","doi-asserted-by":"publisher","first-page":"1842","DOI":"10.1002\/spe.2870","volume":"50","author":"R Amankwah","year":"2020","unstructured":"Amankwah R, Chen J, Kudjo PK, Towey D (2020) An empirical comparison of commercial and open-source web vulnerability scanners. Softw - Pract Exp 50(9):1842\u20131857","journal-title":"Softw - Pract Exp"},{"key":"10179_CR4","unstructured":"Anderson T (2020) Linux in 2020: 27.8 million lines of code in the kernel, 1.3 million in systemd. The Register URL https:\/\/www.theregister.com\/2020\/01\/06\/linux_2020_kernel_systemd_code\/. Accessed 21 Dec 2021"},{"key":"10179_CR5","doi-asserted-by":"crossref","unstructured":"Antunes N, Vieira M (2009) Comparing the effectiveness of penetration testing and static code analysis on the detection of sql injection vulnerabilities in web services. In: 2009 15th IEEE Pacific Rim International Symposium on Dependable Computing. IEEE, pp 301\u2013306","DOI":"10.1109\/PRDC.2009.54"},{"key":"10179_CR6","doi-asserted-by":"crossref","unstructured":"Antunes N, Vieira M (2010) Benchmarking vulnerability detection tools for web services. In: 2010 IEEE International Conference on Web Services,IEEE, pp. 203\u2013210","DOI":"10.1109\/ICWS.2010.76"},{"issue":"7","key":"10179_CR7","doi-asserted-by":"publisher","first-page":"1279","DOI":"10.1016\/j.infsof.2012.11.007","volume":"55","author":"A Austin","year":"2013","unstructured":"Austin A, Holmgreen C, Williams L (2013) A comparison of the efficiency and effectiveness of vulnerability discovery techniques. Inf Softw Technol 55(7):1279\u20131288","journal-title":"Inf Softw Technol"},{"key":"10179_CR8","doi-asserted-by":"crossref","unstructured":"Austin A, Williams L (2011) One technique is not enough: A comparison of vulnerability discovery techniques. In: 2011 International Symposium on Empirical Software Engineering and Measurement, IEEE, pp. 97\u2013106","DOI":"10.1109\/ESEM.2011.18"},{"key":"10179_CR9","unstructured":"Bannister A (2021) Healthcare provider texas ent alerts 535,000 patients to data breach. The Daily Swig. [Online; Publication Date 20 Dec 2021; Accessed 21 Dec 2021]"},{"issue":"901","key":"10179_CR10","first-page":"268","volume":"160","author":"MS Bartlett","year":"1937","unstructured":"Bartlett MS (1937) Properties of sufficiency and statistical tests. Proc R Soc A: Math Phys Eng Sci 160(901):268\u2013282","journal-title":"Proc R Soc A: Math Phys Eng Sci"},{"key":"10179_CR11","unstructured":"Bau J, Wang F, Bursztein E, Mutchler P, Mitchell JC (2012) Vulnerability factors in new web applications: Audit tools, developer selection & languages. Tech. rep., Stanford, https:\/\/seclab.stanford.edu\/websec\/scannerPaper.pdf. Accessed 21 Dec, 2021"},{"key":"10179_CR12","unstructured":"Campbell GA (2020) What is \u2019taint analysis\u2019 and why do i care?, https:\/\/blog.sonarsource.com\/what-is-taint-analysis"},{"key":"10179_CR13","unstructured":"Cass S (2021) Top programming languages 2021. IEEE Spectr, https:\/\/spectrum.ieee.org\/top-programming-languages-2021. Accessed 21 Dec 2021"},{"key":"10179_CR14","unstructured":"Cass S, Kulkarni P, Guizzo E (2021) Interactive: Top Programming Languages 2021. IEEE Spectrum, https:\/\/spectrum.ieee.org\/top-programming-languages\/. Accessed 20 Apr 2022"},{"issue":"3","key":"10179_CR15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.4018\/IJSSSP.2018070101","volume":"9","author":"ML Chaim","year":"2018","unstructured":"Chaim ML, Santos DS, Cruzes DS (2018) What do we know about buffer overflow detection?: A survey on techniques to detect a persistent vulnerability. International Journal of Systems and Software Security and Protection (IJSSSP) 9(3):1\u201333","journal-title":"International Journal of Systems and Software Security and Protection (IJSSSP)"},{"issue":"6","key":"10179_CR16","doi-asserted-by":"publisher","first-page":"551","DOI":"10.1016\/0895-4356(90)90159-M","volume":"43","author":"DV Cicchetti","year":"1990","unstructured":"Cicchetti DV, Feinstein AR (1990) High agreement but low kappa: Ii. resolving the paradoxes. J Clin Epidemiol 43(6):551\u2013558","journal-title":"J Clin Epidemiol"},{"issue":"1","key":"10179_CR17","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1177\/001316446002000104","volume":"20","author":"J Cohen","year":"1960","unstructured":"Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37\u201346","journal-title":"Educ Psychol Meas"},{"key":"10179_CR18","unstructured":"Condon C, Miller H (2021) Maryland health department says there\u2019s no evidence of data lost after cyberattack; website is back online. Baltimore Sun, https:\/\/www.baltimoresun.com\/health\/bs-hs-mdh-website-down-20211206-o2ky2sn5znb3pdwtnu2a7m5g6q-story.html. Accessed 21 Dec 2021"},{"key":"10179_CR19","volume-title":"Quasi-experimentation: Design and analysis issues for field settings","author":"TD Cook","year":"1979","unstructured":"Cook TD, Campbell DT (1979) Quasi-experimentation: Design and analysis issues for field settings. Rand McNally College Publishing, Chicago"},{"key":"10179_CR20","doi-asserted-by":"publisher","DOI":"10.4135\/9781452230153","volume-title":"Basics of qualitative research: Techniques and procedures for developing grounded theory","author":"J Corbin","year":"2008","unstructured":"Corbin J, Strauss A (2008) Basics of qualitative research: Techniques and procedures for developing grounded theory, 3rd edn. SAGE Publications Inc., California","edition":"3rd edn."},{"key":"10179_CR21","unstructured":"Cowan C, Wagle F, Pu C, Beattie S, Walpole J (2000) Buffer overflows: Attacks and defenses for the vulnerability of the decade. In: Proceedings DARPA Information Survivability Conference and Exposition. DISCEX\u201900, IEEE, vol. 2, pp. 119\u2013129"},{"key":"10179_CR22","doi-asserted-by":"crossref","unstructured":"Cruzes DS, Felderer M, Oyetoyan TD, Gander M, Pekaric I (2017) How is security testing done in agile teams? a cross-case analysis of four software teams. In: International Conference on Agile Software Development, Springer, Cham, pp. 201\u2013216","DOI":"10.1007\/978-3-319-57633-6_13"},{"key":"10179_CR23","doi-asserted-by":"crossref","unstructured":"Dambra S, Bilge L, Balzarotti D (2020) Sok: Cyber insurance\u2013technical challenges and a system security roadmap. In: 2020 IEEE Symposium on Security and Privacy (SP), IEEE, pp. 1367\u20131383","DOI":"10.1109\/SP40000.2020.00019"},{"issue":"3","key":"10179_CR24","doi-asserted-by":"publisher","first-page":"319","DOI":"10.2307\/249008","volume":"13","author":"FD Davis","year":"1989","unstructured":"Davis FD (1989) Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly 13(3):319\u2013340","journal-title":"MIS Quarterly"},{"key":"10179_CR25","doi-asserted-by":"publisher","unstructured":"Delaitre AM, Stivalet BC, Black PE, Okun V, Cohen TS, Ribeiro A (2018) Sate v report: Ten years of static analysis tool expositions. NIST SP 500-326, National Institute of Standards and Technology (NIST), https:\/\/doi.org\/10.6028\/NIST.SP.500-326. Accessed 20 Jul 2021","DOI":"10.6028\/NIST.SP.500-326"},{"key":"10179_CR26","unstructured":"Desjardins J (2017) Here\u2019s how many millions of lines of code it takes to run different software. Business Insider, https:\/\/www.businessinsider.com\/how-many-lines-of-code-it-takes-to-run-different-software-2017-2. Accessed 21 Dec 2021"},{"key":"10179_CR27","doi-asserted-by":"crossref","unstructured":"Doup\u00e9 A, Cova M, Vigna G (2010) Why johnny can\u2019t pentest: An analysis of black-box web vulnerability scanners. In: International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment, Springer, pp. 111\u2013131","DOI":"10.1007\/978-3-642-14215-4_7"},{"key":"10179_CR28","doi-asserted-by":"crossref","unstructured":"Elder SE, Zahan N, Kozarev V, Shu R, Menzies T, Williams L (2021) Structuring a comprehensive software security course around the owasp application security verification standard. In: 2021 IEEE\/ACM 43rd International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET), IEEE, pp. 95\u2013104","DOI":"10.1109\/ICSE-SEET52601.2021.00019"},{"key":"10179_CR29","unstructured":"Epic Systems Corporation (2020) From healthcare to mapping the milky way: 5 things you didn\u2019t know about epic\u2019s tech, https:\/\/www.epic.com\/epic\/post\/healthcare-mapping-milky-way-5-things-didnt-know-epics-tech. Accessed 07 Dec 2021"},{"key":"10179_CR30","unstructured":"Executive Order 14028 (2021) Executive order on improving the nation\u2019s cybersecurity. Exec. Order No. 14028, 86 FR 26633, https:\/\/www.federalregister.gov\/d\/2021-10460"},{"issue":"6","key":"10179_CR31","doi-asserted-by":"publisher","first-page":"543","DOI":"10.1016\/0895-4356(90)90158-L","volume":"43","author":"AR Feinstein","year":"1990","unstructured":"Feinstein AR, Cicchetti DV (1990) High agreement but low kappa: I. the problems of two paradoxes. J Clin Epidemiol 43(6):543\u2013549","journal-title":"J Clin Epidemiol"},{"key":"10179_CR32","unstructured":"Feldt R, Magazinius A (2010) Validity threats in empirical software engineering research-an initial survey.. In: Seke, pp. 374\u2013379"},{"issue":"5","key":"10179_CR33","doi-asserted-by":"publisher","first-page":"2959","DOI":"10.1007\/s11135-012-9745-9","volume":"47","author":"GC Feng","year":"2013","unstructured":"Feng GC (2013) Factors affecting intercoder reliability: A monte carlo experiment. Qual Quant 47(5):2959\u20132982","journal-title":"Qual Quant"},{"key":"10179_CR34","doi-asserted-by":"publisher","unstructured":"Fielding RT, Reschke J (2014) Hypertext Transfer Protocol (HTTP\/1.1): Semantics and Content. RFC Editor https:\/\/doi.org\/10.7231\/RFC7231, https:\/\/rfc-editor.org\/rfc\/rfc7231.txt. Accessed 21 Dec 2021","DOI":"10.7231\/RFC7231"},{"key":"10179_CR35","unstructured":"Finifter M, Akhawe D, Wagner D (2013) An empirical study of vulnerability rewards programs. In: 22nd USENIX Security Symposium (USENIX Security 13), USENIX, Washington, D.C., pp. 273\u2013288"},{"key":"10179_CR36","doi-asserted-by":"crossref","unstructured":"Fonseca J, Vieira M, Madeira H (2007) Testing and comparing web vulnerability scanning tools for sql injection and xss attacks. In: 13th Pacific Rim international symposium on dependable computing (PRDC 2007), IEEE, pp. 365\u2013372","DOI":"10.1109\/PRDC.2007.55"},{"issue":"2","key":"10179_CR37","first-page":"113","volume":"1","author":"PA Games","year":"1976","unstructured":"Games PA, Howell JF (1976) Pairwise multiple comparison procedures with unequal n\u2019s and\/or variances: a monte carlo study. J Educ Stat 1(2):113\u2013125","journal-title":"J Educ Stat"},{"key":"10179_CR38","unstructured":"Github (2021) The 2021 State of the Octoverse, https:\/\/octoverse.github.com\/. Accessed 20 Apr 2022"},{"key":"10179_CR39","doi-asserted-by":"crossref","unstructured":"Gon\u00e7ales L, Farias K, da Silva BC (2021) Measuring the cognitive load of software developers: An extended systematic mapping study. Inf Softw Technol 106563","DOI":"10.1016\/j.infsof.2021.106563"},{"issue":"5","key":"10179_CR40","doi-asserted-by":"publisher","first-page":"1920","DOI":"10.1007\/s10664-015-9403-7","volume":"21","author":"M Hafiz","year":"2016","unstructured":"Hafiz M, Fang M (2016) Game of detections: how are security vulnerabilities discovered in the wild?. Empir Softw Eng 21(5):1920\u20131959","journal-title":"Empir Softw Eng"},{"key":"10179_CR41","doi-asserted-by":"crossref","unstructured":"Imtiaz N, Rahman A, Farhana E, Williams L (2019) Challenges with responding to static analysis tool alerts. In: 2019 IEEE\/ACM 16th International Conference on Mining Software Repositories (MSR), IEEE, pp. 245\u2013249","DOI":"10.1109\/MSR.2019.00049"},{"key":"10179_CR42","unstructured":"ISO\/IEC\/IEEE (2013) Software and systems engineering \u2014 software testing \u2014 part 1: concepts and definitions. ISO\/IEC\/IEEE 29119-1:2013, International Organization for Standardization (ISO), International Electrotechnical Commission (IES), and Institute of Electrical and Electronics Engineers (IEEE)"},{"issue":"2","key":"10179_CR43","doi-asserted-by":"publisher","first-page":"303","DOI":"10.1007\/s10664-013-9266-8","volume":"19","author":"J Itkonen","year":"2014","unstructured":"Itkonen J, M\u00e4ntyl\u00e4 MV (2014) Are test cases needed? replicated comparison between exploratory and test-case-based software testing. Empir Softw Eng 19(2):303\u2013342","journal-title":"Empir Softw Eng"},{"issue":"5","key":"10179_CR44","doi-asserted-by":"publisher","first-page":"707","DOI":"10.1109\/TSE.2012.55","volume":"39","author":"J Itkonen","year":"2013","unstructured":"Itkonen J, M\u00e4ntyl\u00e4 MV, Lassenius C (2013) The role of the tester\u2019s knowledge in exploratory software testing. IEEE Trans Softw Eng 39(5):707\u2013724","journal-title":"IEEE Trans Softw Eng"},{"key":"10179_CR45","doi-asserted-by":"crossref","unstructured":"Johnson B, Song Y, Murphy-Hill E, Bowdidge R (2013) Why don\u2019t software developers use static analysis tools to find bugs?. In: Proceedings of the 2013 International Conference on Software Engineering, IEEE Press, pp. 672\u2013681","DOI":"10.1109\/ICSE.2013.6606613"},{"key":"10179_CR46","doi-asserted-by":"publisher","unstructured":"Joint Task Force Transformation Initiative (2013) Security and privacy controls for federal information systems and organizations. NIST SP 800-53, National Institute of Standards and Technology (NIST), https:\/\/doi.org\/10.6028\/NIST.SP.800-53r4. Accessed 20 Jul 2021","DOI":"10.6028\/NIST.SP.800-53r4"},{"key":"10179_CR47","doi-asserted-by":"publisher","DOI":"10.4135\/9781483384733","volume-title":"Experimental design: Procedures for the behavioral sciences","author":"R Kirk","year":"2013","unstructured":"Kirk R (2013) Experimental design: Procedures for the behavioral sciences, 4th edn. Sage Publications, Thousand Oaks","edition":"4th edn."},{"issue":"2","key":"10179_CR48","doi-asserted-by":"publisher","first-page":"579","DOI":"10.1007\/s10664-016-9437-5","volume":"22","author":"B Kitchenham","year":"2017","unstructured":"Kitchenham B, Madeyski L, Budgen D, Keung J, Brereton P, Charters S, Gibbs S, Pohthong A (2017) Robust statistical methods for empirical software engineering. Empir Softw Eng 22(2):579\u2013630","journal-title":"Empir Softw Eng"},{"key":"10179_CR49","doi-asserted-by":"publisher","DOI":"10.1201\/b19467","volume-title":"Evidence-based software engineering and systematic reviews, vol 4","author":"BA Kitchenham","year":"2015","unstructured":"Kitchenham BA, Budgen D, Brereton P (2015) Evidence-based software engineering and systematic reviews, vol 4. CRC press, Boca Raton"},{"key":"10179_CR50","doi-asserted-by":"crossref","unstructured":"Klees G, Ruef A, Cooper B, Wei S, Hicks M (2018) Evaluating fuzz testing. In: Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pp. 2123\u20132138","DOI":"10.1145\/3243734.3243804"},{"key":"10179_CR51","doi-asserted-by":"publisher","first-page":"182004","DOI":"10.1109\/ACCESS.2019.2960449","volume":"7","author":"M Liu","year":"2019","unstructured":"Liu M, Zhang B, Chen W, Zhang X (2019) A survey of exploitation and detection methods of xss vulnerabilities. IEEE Access 7:182004\u2013182016","journal-title":"IEEE Access"},{"issue":"4","key":"10179_CR52","doi-asserted-by":"publisher","first-page":"587","DOI":"10.1111\/j.1468-2958.2002.tb00826.x","volume":"28","author":"M Lombard","year":"2002","unstructured":"Lombard M, Snyder-Duch J, Bracken CC (2002) Content analysis in mass communication: Assessment and reporting of intercoder reliability. Hum Commun Res 28(4):587\u2013604","journal-title":"Hum Commun Res"},{"key":"10179_CR53","doi-asserted-by":"crossref","unstructured":"Lung J, Aranda J, Easterbrook S, Wilson G (2008) On the difficulty of replicating human subjects studies in software engineering. In: 2008 ACM\/IEEE 30th International Conference on Software Engineering, pp. 191\u2013200","DOI":"10.1145\/1368088.1368115"},{"key":"10179_CR54","unstructured":"Mallet F (2016) Sonaranalyzer for java: Tricky bugs are running scared. https:\/\/blog.sonarsource.com\/sonaranalyzer-for-java-tricky-bugs-are-running-scared, Accessed 05 Dec 2021"},{"key":"10179_CR55","volume-title":"Software security: building security in","author":"G McGraw","year":"2006","unstructured":"McGraw G (2006) Software security: building security in. Addison-Wesley Professional, Boston"},{"key":"10179_CR56","unstructured":"MITRE (2016) Common vulnerabilities and exposures (cve) numbering authority (cna) rules. https:\/\/cve.mitre.org\/cve\/cna\/CNA_Rules_v1.1.pdf, Accessed 24 July 2021"},{"key":"10179_CR57","unstructured":"MITRE (2021) Cve \u2192 cwe mapping guidance. In: (MITRE 2021b), https:\/\/cwe.mitre.org\/documents\/cwe_usage\/guidance.html. Accessed 24 Jul 2021"},{"key":"10179_CR58","unstructured":"MITRE (2021) Cwe common weakness enumeration (website), https:\/\/cwe.mitre.org\/. Accessed 20 Jul 2021"},{"key":"10179_CR59","unstructured":"MITRE (2021) Cwe view: Weaknesses in owasp top ten (2021). In: (MITRE 2021b), https:\/\/cwe.mitre.org\/data\/definitions\/1344.html. Accessed 09 Dec 2021"},{"key":"10179_CR60","unstructured":"MITRE (2022) Cwe 1003 - cwe view: Weaknesses for simplified mapping of published vulnerabilities"},{"key":"10179_CR61","doi-asserted-by":"publisher","first-page":"146","DOI":"10.1016\/j.infsof.2018.05.011","volume":"102","author":"P Morrison","year":"2018","unstructured":"Morrison P, Moye D, Pandita R, Williams L (2018) Mapping the field of software life cycle security metrics. Inf Softw Technol 102:146\u2013159","journal-title":"Inf Softw Technol"},{"key":"10179_CR62","unstructured":"Mozilla (2021) Http messages, https:\/\/developer.mozilla.org\/en-US\/docs\/Web\/HTTP\/Messages. Accessed 21 Dec 2021"},{"key":"10179_CR63","doi-asserted-by":"crossref","unstructured":"Nagarakatte S, Zhao J, Martin Milo MK, Zdancewic S (2009) Softbound: Highly compatible and complete spatial memory safety for c. In: Proceedings of the 30th ACM SIGPLAN Conference on Programming Language Design and Implementation, pp 245\u2013258","DOI":"10.1145\/1543135.1542504"},{"key":"10179_CR64","unstructured":"NVD (2021) Cwe over time. In: (NVD 2021b), https:\/\/nvd.nist.gov\/general\/visualizations\/vulnerability-visualizations\/cwe-over-time. Accessed 05 Dec 2021"},{"key":"10179_CR65","unstructured":"NVD (2021) National vulnerability database (website). National Institute of Standards and Technology (NIST), https:\/\/nvd.nist.gov\/. Accessed 01 Nov 2021"},{"key":"10179_CR66","unstructured":"NVD (2021) Nvd - general faqs. National Institute of Standards and Technology (NIST). In: (NVD 2021b) https:\/\/nvd.nist.gov\/general\/FAQ-Sections\/General-FAQs. Accessed 04 Apr 2022"},{"key":"10179_CR67","unstructured":"NVD (2021) Vulnerabilities. In: (NVD 2021b), https:\/\/nvd.nist.gov\/vuln. Accessed 01 Nov 2021"},{"key":"10179_CR68","doi-asserted-by":"publisher","unstructured":"Okun V, Delaitre A, Black PE (2010) The second static analysis tool exposition (sate) 2009. NIST SP 500-287, National Institute of Standards and Technology (NIST), https:\/\/doi.org\/10.6028\/NIST.SP.500-287. Accessed 20 Jul 2021","DOI":"10.6028\/NIST.SP.500-287"},{"key":"10179_CR69","doi-asserted-by":"publisher","unstructured":"Okun V, Delaitre A, Black PE (2011) Report on the third static analysis tool exposition (sate 2010). NIST SP 500-283, National Institute of Standards and Technology (NIST), https:\/\/doi.org\/10.6028\/NIST.SP.500-283. Accessed 20 Jul 2021","DOI":"10.6028\/NIST.SP.500-283"},{"key":"10179_CR70","doi-asserted-by":"publisher","unstructured":"Okun V, Delaitre A, Black PE (2013) Report on the static analysis tool exposition (sate) iv. NIST SP 500-297, National Institute of Standards and Technology (NIST), https:\/\/doi.org\/10.6028\/NIST.SP.500-297. Accessed 20 Jul 2021","DOI":"10.6028\/NIST.SP.500-297"},{"key":"10179_CR71","doi-asserted-by":"publisher","unstructured":"Okun V, Gaucher R, Black PE (2009) Static analysis tool exposition (sate) 2008. NIST SP 500-279, National Institute of Standards and Technology (NIST), https:\/\/doi.org\/10.6028\/NIST.SP.500-279. Accessed 20 Jul 2021","DOI":"10.6028\/NIST.SP.500-279"},{"key":"10179_CR72","unstructured":"Open Web Application Security Project (OWASP) Foundation (2013) Owasp top ten - 2010, https:\/\/owasp.org\/www-pdf-archive\/OWASP_Top_10_-_2010.pdf. Accessed 05 Dec 2021"},{"key":"10179_CR73","unstructured":"Open Web Application Security Project (OWASP) Foundation (2013) Owasp top ten - 2013, https:\/\/owasp.org\/www-pdf-archive\/OWASP_Top_10_-_2013.pdf. Accessed 05 Dec 2021"},{"key":"10179_CR74","unstructured":"Open Web Application Security Project (OWASP) Foundation (2017) Owasp top ten - 2017, https:\/\/owasp.org\/www-project-top-ten\/2017\/. Accessed 05 Dec 2021"},{"key":"10179_CR75","unstructured":"Open Web Application Security Project (OWASP) Foundation (2021) Owasp top ten - 2021, https:\/\/owasp.org\/Top10\/. Accessed 05 Dec 2021"},{"key":"10179_CR76","unstructured":"Open Web Application Security Project (OWASP) Foundation (2021) The owasp top ten application security risks project, https:\/\/owasp.org\/www-project-top-ten\/. Accessed 09 Dec 2021"},{"key":"10179_CR77","unstructured":"Open Web Application Security Project (OWASP) Foundation (2021) Owasp zap, https:\/\/www.zaproxy.org\/. Accessed: 21-Dec-2021"},{"key":"10179_CR78","unstructured":"OpenMRS (2020) Openmrs developer manual, http:\/\/devmanual.openmrs.org\/en\/. Accessed 24 Jul 2021"},{"key":"10179_CR79","unstructured":"OpenMRSAtlas (2021) Openmrs atlas, https:\/\/atlas.openmrs.org\/. Accessed 24 Jul 2021"},{"key":"10179_CR80","unstructured":"OWASP ZAP Dev Team (2021) Getting started - features - alerts. In: (Team OZD 2021), https:\/\/www.zaproxy.org\/docs\/desktop\/start\/features\/alerts\/. Accessed 06 Dec 2021"},{"key":"10179_CR81","unstructured":"OWASP ZAP Dev Team (2021) Getting started - features - spider. In: (Team OZD 2021), https:\/\/www.zaproxy.org\/docs\/desktop\/start\/features\/spider\/. Accessed 20 Jul 2021"},{"key":"10179_CR82","doi-asserted-by":"crossref","unstructured":"Pfahl D, Yin H, M\u00e4ntyl\u00e4 MV, M\u00fcnch J (2014) How is exploratory testing used? a state-of-the-practice survey. In: Proceedings of the 8th ACM\/IEEE international symposium on empirical software engineering and measurement, ACM, p. 5","DOI":"10.1145\/2652524.2652531"},{"key":"10179_CR83","doi-asserted-by":"crossref","unstructured":"Purkayastha S, Goyal S, Phillips T, Wu H, Haakenson B, Zou X (2020) Continuous security through integration testing in an electronic health records system. In: 2020 International Conference on Software Security and Assurance (ICSSA), IEEE, pp. 26\u201331","DOI":"10.1109\/ICSSA51305.2020.00012"},{"key":"10179_CR84","unstructured":"Radio New Zealand (RNZ) (2021) Health ministry announces $75m to plug cybersecurity gaps, https:\/\/www.rnz.co.nz\/news\/national\/458331\/health-ministry-announces-75m-to-plug-cybersecurity-gaps. Accessed 21 Dec 2021"},{"key":"10179_CR85","doi-asserted-by":"crossref","unstructured":"Rahman AAU, Helms E, Williams L, Parnin C (2015) Synthesizing continuous deployment practices used in software development. In: 2015 Agile Conference, IEEE, pp. 1\u201310","DOI":"10.1109\/Agile.2015.12"},{"key":"10179_CR86","doi-asserted-by":"crossref","unstructured":"Ralph P, Tempero E (2018) Construct validity in software engineering research and software metrics. In: Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering 2018, pp. 13\u201323","DOI":"10.1145\/3210459.3210461"},{"issue":"1","key":"10179_CR87","first-page":"21","volume":"2","author":"NM Razali","year":"2011","unstructured":"Razali NM, Wah Y B, et al (2011) Power comparisons of shapiro-wilk, kolmogorov-smirnov, lilliefors and anderson-darling tests. J Stat Modelling Anal 2(1):21\u201333","journal-title":"J Stat Modelling Anal"},{"key":"10179_CR88","doi-asserted-by":"crossref","unstructured":"Scandariato R, Walden J, Joosen W (2013) Static analysis versus penetration testing: A controlled experiment. In: 2013 IEEE 24th international symposium on software reliability engineering (ISSRE), IEEE, pp. 451\u2013460","DOI":"10.1109\/ISSRE.2013.6698898"},{"key":"10179_CR89","unstructured":"Scanlon T (2018) 10 types of application security testing tools: When and how to use them. Blog, Software Engineering Institute, Carnegie Mellon University, https:\/\/insights.sei.cmu.edu\/blog\/10-types-of-application-security-testing-tools-when-and-how-to-use-them. Accessed 20 Jul 2021"},{"key":"10179_CR90","doi-asserted-by":"crossref","unstructured":"Smith B, Williams L (2012) On the effective use of security test patterns. In: 2012 IEEE Sixth International Conference on Software Security and Reliability, IEEE, pp. 108\u2013117","DOI":"10.1109\/SERE.2012.23"},{"key":"10179_CR91","doi-asserted-by":"crossref","unstructured":"Smith B, Williams LA (2011) Systematizing security test planning using functional requirements phrases. Tech. Rep. TR-2011-5, North Carolina State University. Dept. of Computer Science","DOI":"10.1145\/1985793.1986019"},{"key":"10179_CR92","unstructured":"Smith J, Do LNQ, Murphy-Hill E (2020) Why can\u2019t johnny fix vulnerabilities: A usability evaluation of static analysis tools for security. In: Sixteenth Symposium on Usable Privacy and Security ({SOUPS} 2020), USENIX, pp. 221\u2013238"},{"key":"10179_CR93","doi-asserted-by":"crossref","unstructured":"Smith J, Johnson B, Murphy-Hill E, Chu B, Lipford HR (2015) Questions developers ask while diagnosing potential security vulnerabilities with static analysis. In: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ACM, pp. 248\u2013259","DOI":"10.1145\/2786805.2786812"},{"key":"10179_CR94","unstructured":"SonarSource (2019) Sonarqube documentation: Security-related rules, https:\/\/docs.sonarqube.org\/8.2\/user-guide\/security-rules\/. Accessed 06 Dec 2021"},{"key":"10179_CR95","unstructured":"StackOverflow (2021) 2021 Developer Survey, https:\/\/insights.stackoverflow.com\/survey\/2021#technology-most-popular-technologies. Accessed: 07 Dec 2021"},{"key":"10179_CR96","unstructured":"Team OZD (ed.) (2021) The owasp zed attack proxy (zap) desktop user guide"},{"key":"10179_CR97","doi-asserted-by":"crossref","unstructured":"T\u00f8ndel IA, Jaatun MG, Cruzes DS, Williams L (2019) Collaborative security risk estimation in agile software development. Information & Computer Security 27(4)","DOI":"10.1108\/ICS-12-2018-0138"},{"key":"10179_CR98","unstructured":"U.S. Cybersecurity and Infrastructure Security Agency (CISA) (2021) Provide medical care is in critical condition: Analysis and stakeholder decision support to minimize further harm, https:\/\/www.cisa.gov\/sites\/default\/files\/publications\/Insights_MedicalCare_FINAL-v2_0.pdf. Accessed 21 Dec 2021"},{"key":"10179_CR99","unstructured":"US Dept of Veterans Affairs, Office of Information and Technology, Enterprise Program Management Office (2021) VA Monograph, https:\/\/www.va.gov\/vdl\/documents\/Monograph\/Monograph\/VistA_Monograph_0421_REDACTED.pdf. Accessed 07 Dec 2021"},{"key":"10179_CR100","unstructured":"van der Stock A, Cuthbert D, Manico J, Grossman J C, Burnett M (2019) Application security verification standard. Rev. 4.0.1, Open Web Application Security Project (OWASP), https:\/\/github.com\/OWASP\/ASVS\/tree\/v4.0.1\/4.0. Accessed 20 Jul 2021"},{"key":"10179_CR101","doi-asserted-by":"crossref","unstructured":"Votipka D, Stevens R, Redmiles E, Hu J, Mazurek M (2018) Hackers vs. testers: A comparison of software vulnerability discovery processes. In: 2018 IEEE Symposium on Security and Privacy (SP), IEEE, pp. 374\u2013391","DOI":"10.1109\/SP.2018.00003"},{"issue":"3","key":"10179_CR102","doi-asserted-by":"publisher","first-page":"254","DOI":"10.1037\/1082-989X.8.3.254","volume":"8","author":"RR Wilcox","year":"2003","unstructured":"Wilcox RR, Keselman HJ (2003) Modern robust data analysis methods: measures of central tendency. Psychol Methods 8(3):254","journal-title":"Psychol Methods"},{"key":"10179_CR103","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-29044-2","volume-title":"Experimentation in software engineering","author":"C Wohlin","year":"2012","unstructured":"Wohlin C, Runeson P, H\u00f6st M, Ohlsson MC, Regnell B, Wessl\u00e9n A (2012) Experimentation in software engineering. Springer Science & Business Media, New York"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-022-10179-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-022-10179-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-022-10179-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,10,14]],"date-time":"2022-10-14T12:11:37Z","timestamp":1665749497000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-022-10179-6"}},"subtitle":["An empirical case study comparing vulnerability detection techniques on a Java application"],"short-title":[],"issued":{"date-parts":[[2022,8,6]]},"references-count":103,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2022,11]]}},"alternative-id":["10179"],"URL":"https:\/\/doi.org\/10.1007\/s10664-022-10179-6","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,8,6]]},"assertion":[{"value":"20 May 2022","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 August 2022","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}],"article-number":"154"}}