{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T03:28:47Z","timestamp":1781062127912,"version":"3.54.1"},"reference-count":46,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2023,9,1]],"date-time":"2023-09-01T00:00:00Z","timestamp":1693526400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,9,12]],"date-time":"2023-09-12T00:00:00Z","timestamp":1694476800000},"content-version":"vor","delay-in-days":11,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"European Society of Regional Fund","award":["ERDF-0801379r"],"award-info":[{"award-number":["ERDF-0801379r"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2023,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The use of static analysis security testing (SAST) tools has been increasing in recent years. However, previous studies have shown that, when shipped to end users such as development or security teams, the findings of these tools are often unsatisfying. Users report high numbers of false positives or long analysis times, making the tools unusable in the daily workflow. To address this, SAST tool creators provide a wide range of configuration options, such as customization of rules through domain-specific languages or specification of the application-specific analysis scope. In this paper, we study the configuration space of selected existing SAST tools when used within the integrated development environment (IDE). We focus on the configuration options that impact three dimensions, for which a trade-off is unavoidable, i.e., precision, recall, and analysis runtime. We perform a between-subjects user study with 40 users from multiple development and security teams - to our knowledge, the largest population for this kind of user study in the software engineering community. The results show that users who configure SAST tools are more effective in resolving security vulnerabilities detected by the tools than those using the default configuration. Based on post-study interviews, we identify common strategies that users have while configuring the SAST tools to provide further insights for tool creators. Finally, an evaluation of the configuration options of two commercial SAST tools, <jats:sc>Fortify<\/jats:sc> and <jats:sc>CheckMarx<\/jats:sc>, reveals that a quarter of the users do not understand the configuration options provided. The configuration options that are found most useful relate to the analysis scope.<\/jats:p>","DOI":"10.1007\/s10664-023-10354-3","type":"journal-article","created":{"date-parts":[[2023,9,12]],"date-time":"2023-09-12T10:02:50Z","timestamp":1694512970000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Can the configuration of static analyses make resolving security vulnerabilities more effective? - A user study"],"prefix":"10.1007","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4424-5838","authenticated-orcid":false,"given":"Goran","family":"Piskachev","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthias","family":"Becker","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eric","family":"Bodden","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,9,12]]},"reference":[{"issue":"6","key":"10354_CR1","doi-asserted-by":"publisher","first-page":"259","DOI":"10.1145\/2666356.2594299","volume":"49","author":"S Arzt","year":"2014","unstructured":"Arzt S, Rasthofer S, Fritz C, Bodden E, Bartel A, Klein J, Le Traon Y, Octeau D, McDaniel P (2014) Flowdroid: Precise context, flow, field, object-sensitive and lifecycle-aware taint analysis for android apps. Acm Sigplan Notices 49(6):259\u2013269","journal-title":"Acm Sigplan Notices"},{"key":"10354_CR2","unstructured":"Benchmark O (2021) Owasp. https:\/\/owasp.org\/www-project-benchmark\/, online; Accessed January 2021"},{"key":"10354_CR3","doi-asserted-by":"publisher","unstructured":"Charness G, Gneezy U, Kuhn MA (2012) Experimental methods: Between-subject and within-subject design. J Econ Behav Organ 81(1):1\u20138. https:\/\/doi.org\/10.1016\/j.jebo.2011.08.00, https:\/\/ideas.repec.org\/a\/eee\/jeborg\/v81y2012i1p1-8.html","DOI":"10.1016\/j.jebo.2011.08.00"},{"key":"10354_CR4","unstructured":"Checkmarx (2021) Checkmarx. Online; Accessed January 2021"},{"key":"10354_CR5","doi-asserted-by":"crossref","unstructured":"Christakis M, Bird C (2016) What developers want and need from program analysis: An empirical study. In: Proceedings of the 31st IEEE\/ACM International Conference on Automated Software Engineering, ACM, New York, NY, USA, ASE 2016, pp 332\u2013343","DOI":"10.1145\/2970276.2970347"},{"key":"10354_CR6","volume-title":"Quasi-Experimentation: Design and Analysis Issues for Field Settings","author":"TD Cook","year":"1979","unstructured":"Cook TD, Campbell DT (1979) Quasi-Experimentation: Design and Analysis Issues for Field Settings. Houghton Mifflin, Boston, USA, Boston"},{"key":"10354_CR7","unstructured":"Facebook (2021) Infer. Online; Accessed January 2021"},{"key":"10354_CR8","unstructured":"Github S (2021) Lgtm. Online; Accessed January 2021"},{"key":"10354_CR9","doi-asserted-by":"crossref","unstructured":"Gordon M, deokhwan K, Perkins J, Gilham L, Nguyen N, Rinard M (2015) Information-flow analysis of android applications in droidsafe. In: Network and Distributed System Security Symposium 2015, 10.14722\/ndss.2015.23089","DOI":"10.14722\/ndss.2015.23089"},{"key":"10354_CR10","unstructured":"Grammatech (2021) Codesonar. Online; Accessed January 2021"},{"key":"10354_CR11","doi-asserted-by":"publisher","unstructured":"Habib A, Pradel M (2018) How many of all bugs do we find? a study of static bug detectors. In: Proceedings of the 33rd ACM\/IEEE International Conference on Automated Software Engineering, Association for Computing Machinery, New York, NY, USA, ASE 2018, p 317\u2013328, https:\/\/doi.org\/10.1145\/3238147.3238213","DOI":"10.1145\/3238147.3238213"},{"key":"10354_CR12","doi-asserted-by":"crossref","unstructured":"Johnson B, Song Y, Murphy-Hill E, Bowdidge R (2013) Why don\u2019t software developers use static analysis tools to find bugs? In: Proceedings of the 2013 International Conference on Software Engineering, IEEE Press, ICSE \u201913, p 672\u2013681","DOI":"10.1109\/ICSE.2013.6606613"},{"key":"10354_CR13","unstructured":"Kr\u00fcger S, Sp\u00e4th J, Ali K, Bodden E, Mezini M (2018) CrySL: An Extensible Approach to Validating the Correct Usage of Cryptographic APIs. In: European Conference on Object-Oriented Programming (ECOOP), pp 10:1\u201310:27, https:\/\/bodden.de\/pubs\/ksa+18crysl.pdf"},{"key":"10354_CR14","doi-asserted-by":"publisher","unstructured":"Lee W, Lee W, Kang D, Heo K, Oh H, Yi K (2017) Sound non-statistical clustering of static analysis alarms. ACM Trans Program Lang Syst 39(4), https:\/\/doi.org\/10.1145\/3095021","DOI":"10.1145\/3095021"},{"key":"10354_CR15","doi-asserted-by":"crossref","unstructured":"Luo L, Bodden E, Sp\u00e4th J (2019) A qualitative analysis of android taint-analysis results. In: 2019 34th IEEE\/ACM International Conference on Automated Software Engineering (ASE), pp 102\u2013114, 10.1109\/ASE.2019.00020","DOI":"10.1109\/ASE.2019.00020"},{"key":"10354_CR16","doi-asserted-by":"publisher","unstructured":"Luo L, Dolby J, Bodden E (2019) MagpieBridge: A General Approach to Integrating Static Analyses into IDEs and Editors (Tool Insights Paper). In: Donaldson AF (ed) 33rd European Conference on Object-Oriented Programming (ECOOP 2019), Schloss Dagstuhl\u2013Leibniz-Zentrum fuer Informatik, Dagstuhl, Germany, Leibniz International Proceedings in Informatics (LIPIcs), vol 134, pp 21:1\u201321:25, https:\/\/doi.org\/10.4230\/LIPIcs.ECOOP.2019.21,http:\/\/drops.dagstuhl.de\/opus\/volltexte\/2019\/10813","DOI":"10.4230\/LIPIcs.ECOOP.2019.21"},{"key":"10354_CR17","unstructured":"of Maryland U (2021) Findbugs. Online; Accessed January 2021"},{"key":"10354_CR18","unstructured":"Microfocus (2021) Fortify. Online; Accessed January 2021"},{"key":"10354_CR19","unstructured":"Mitre CWE (2021a) 2011 cwe\/sans top 25 most dangerous software errors. http:\/\/cwe.mitre.org\/top25\/, online; Accessed January 2021"},{"key":"10354_CR20","unstructured":"Mitre CWE (2021b) Improper neutralization of special elements used in an sql command. https:\/\/cwe.mitre.org\/data\/definitions\/89.html, online; Accessed January 2021"},{"key":"10354_CR21","doi-asserted-by":"publisher","unstructured":"Naiakshina A, Danilova A, Tiefenau C, Herzog M, Dechand S, Smith M (2017) Why do developers get password storage wrong? a qualitative usability study. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, Association for Computing Machinery, New York, NY, USA, CCS 17, pp 311\u2013328, https:\/\/doi.org\/10.1145\/3133956.3134082","DOI":"10.1145\/3133956.3134082"},{"key":"10354_CR22","first-page":"297","volume":"18","author":"A Naiakshina","year":"2018","unstructured":"Naiakshina A, Danilova A, Tiefenau C, Smith M (2018) Deception task design in developer password studies: Exploring a student sample. Proceedings of the Fourteenth USENIX Conference on Usable Privacy and Security, USENIX Association, USA, SOUPS 18:297\u2013313","journal-title":"Proceedings of the Fourteenth USENIX Conference on Usable Privacy and Security, USENIX Association, USA, SOUPS"},{"key":"10354_CR23","doi-asserted-by":"crossref","unstructured":"Naiakshina A, Danilova A, Gerlitz E, von Zezschwitz E, Smith M (2019) If you want, i can store the encrypted password: A password-storage field study with freelance developers. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, New York, NY, USA, CHI 19, pp 1\u201312","DOI":"10.1145\/3290605.3300370"},{"key":"10354_CR24","doi-asserted-by":"crossref","unstructured":"Naiakshina A, Danilova A, Gerlitz E, Smith M (2020) On conducting security developer studies with cs students: Examining a password-storage study with cs students, freelancers, and company developers. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, New York, NY, USA, CHI 20, pp 1\u201313","DOI":"10.1145\/3313831.3376791"},{"key":"10354_CR25","unstructured":"Nguyen Quang Do L, Bodden E (2020) Explaining static analysis with rule graphs. IEEE Transactions on Software Engineering pp 1\u20131, 10.1109\/TSE.2020.3004525"},{"key":"10354_CR26","doi-asserted-by":"publisher","unstructured":"Nguyen Quang Do L, Ali K, Livshits B, Bodden E, Smith J, Murphy-Hill E (2017) Just-in-time static analysis. In: Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis, ACM, New York, NY, USA, ISSTA 2017, pp 307\u2013317, https:\/\/doi.org\/10.1145\/3092703.3092705","DOI":"10.1145\/3092703.3092705"},{"key":"10354_CR27","unstructured":"Nguyen Quang\u00a0Do L, Wright JR, Ali K (2020) Why do software developers use static analysis tools? a user-centered study of developer needs and motivations. In: Proceedings of the Sixteenth Symposium on Usable Privacy and Security, 10.1109\/TSE.2020.3004525"},{"key":"10354_CR28","unstructured":"Patnaik N, Hallett J, Rashid A (2019) Usability smells: An analysis of developers\u2019 struggle with crypto libraries. In: Fifteenth Symposium on Usable Privacy and Security (SOUPS 2019), USENIX Association, Santa Clara, CA, https:\/\/www.usenix.org\/conference\/soups2019\/presentation\/patnaik"},{"key":"10354_CR29","doi-asserted-by":"crossref","unstructured":"Piskachev G, Do LNQ, Bodden E (2019) Codebase-adaptive detection of security-relevant methods. In: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, Association for Computing Machinery, New York, NY, USA, ISSTA 2019, pp 181\u2013191","DOI":"10.1145\/3293882.3330556"},{"key":"10354_CR30","doi-asserted-by":"crossref","unstructured":"Piskachev G, Krishnamurthy R, Bodden E (2021) Secucheck: Engineering configurable taint analysis for software developers. In: 21st IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM 2021), IEEE, Luxembourg","DOI":"10.1109\/SCAM52516.2021.00012"},{"key":"10354_CR31","doi-asserted-by":"publisher","unstructured":"Piskachev G, Sp\u00e4th J, Budde I, Bodden E (2022) Fluently specifying taint-flow queries with fluenttql. Empirical Softw Eng 27(5), https:\/\/doi.org\/10.1007\/s10664-022-10165-y","DOI":"10.1007\/s10664-022-10165-y"},{"key":"10354_CR32","doi-asserted-by":"publisher","unstructured":"Qiu L, Wang Y, Rubin J (2018) Analyzing the analyzers: Flowdroid\/iccta, amandroid, and droidsafe. In: Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, Association for Computing Machinery, New York, NY, USA, ISSTA 2018, pp 176\u2013186, https:\/\/doi.org\/10.1145\/3213846.3213873","DOI":"10.1145\/3213846.3213873"},{"key":"10354_CR33","doi-asserted-by":"crossref","unstructured":"Sadowski C, van Gogh J, Jaspan C, S\u00f6derberg E, Winter C (2015) Tricorder: Building a program analysis ecosystem. In: Proceedings of the 37th International Conference on Software Engineering - Volume 1, IEEE Press, ICSE \u201915, pp 598\u2013608","DOI":"10.1109\/ICSE.2015.76"},{"key":"10354_CR34","doi-asserted-by":"crossref","unstructured":"Siegmund J, Siegmund N, Apel S (2015) Views on internal and external validity in empirical software engineering. In: 2015 IEEE\/ACM 37th IEEE International Conference on Software Engineering, IEEE, vol\u00a01, pp 9\u201319","DOI":"10.1109\/ICSE.2015.24"},{"key":"10354_CR35","doi-asserted-by":"crossref","unstructured":"Smith J, Johnson B, Murphy-Hill E, Chu B, Lipford H (2015) Questions developers ask while diagnosing potential security vulnerabilities with static analysis. In: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ACM, New York, NY, USA, ESEC\/FSE 2015, pp 248\u2013259","DOI":"10.1145\/2786805.2786812"},{"key":"10354_CR36","unstructured":"Smith J, Nguyen Quang\u00a0Do L, Murphy-Hill E (2020) Why can\u2019t johnny fix vulnerabilities: A usability evaluation of static analysis tools for security. In: Proceedings of the Sixteenth Symposium on Usable Privacy and Security, SOUPS 2020"},{"key":"10354_CR37","unstructured":"Snyk (2021) Deepcode. Online; Accessed January 2021"},{"key":"10354_CR38","unstructured":"SonarSource (2021) Sonarqube. Online; Accessed January 2021"},{"key":"10354_CR39","unstructured":"Thomas TW, Lipford H, Chu B, Smith J, Murphy-Hill E (2016) What questions remain? an examination of how developers understand an interactive static analysis tool. In: Twelfth Symposium on Usable Privacy and Security (SOUPS 2016), USENIX Association, Denver, CO, https:\/\/www.usenix.org\/conference\/soups2016\/workshop-program\/wsiw16\/presentation\/thomas"},{"key":"10354_CR40","doi-asserted-by":"crossref","unstructured":"Vassallo C, Panichella S, Palomba F, Proksch S, Zaidman A, Gall HC (2018) Context is king: The developer perspective on the usage of static analysis tools. In: 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER), pp 38\u201349","DOI":"10.1109\/SANER.2018.8330195"},{"key":"10354_CR41","unstructured":"Veracode (2021) Veracode. Online; accessed January 2021"},{"key":"10354_CR42","unstructured":"WebGoat (2021) Owasp webgoat. https:\/\/owasp.org\/www-project-webgoat\/, online; Accessed January 2021"},{"key":"10354_CR43","doi-asserted-by":"publisher","unstructured":"Wei F, Roy S, Ou X, Robby (2018) Amandroid: A precise and general inter-component data flow analysis framework for security vetting of android apps. ACM Trans Priv Secur 21(3),https:\/\/doi.org\/10.1145\/3183575","DOI":"10.1145\/3183575"},{"key":"10354_CR44","doi-asserted-by":"publisher","unstructured":"Witschey J, Zielinska O, Welk A, Murphy-Hill E, Mayhorn C, Zimmermann T (2015) Quantifying developers\u2019 adoption of security tools. In: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ACM, New York, NY, USA, ESEC\/FSE 2015, pp 260\u2013271, https:\/\/doi.org\/10.1145\/2786805.2786816","DOI":"10.1145\/2786805.2786816"},{"key":"10354_CR45","doi-asserted-by":"crossref","unstructured":"Zampetti F, Scalabrino S, Oliveto R, Canfora G, Di Penta M (2017) How open source projects use static code analysis tools in continuous integration pipelines. In: 2017 IEEE\/ACM 14th International Conference on Mining Software Repositories (MSR), pp 334\u2013344","DOI":"10.1109\/MSR.2017.2"},{"key":"10354_CR46","doi-asserted-by":"publisher","unstructured":"Zhang X, Grigore R, Si X, Naik M (2017) Effective interactive resolution of static analysis alarms. Proc ACM Program Lang 1(OOPSLA), https:\/\/doi.org\/10.1145\/3133881","DOI":"10.1145\/3133881"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-023-10354-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-023-10354-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-023-10354-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,2]],"date-time":"2024-04-02T11:18:33Z","timestamp":1712056713000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-023-10354-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9]]},"references-count":46,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2023,9]]}},"alternative-id":["10354"],"URL":"https:\/\/doi.org\/10.1007\/s10664-023-10354-3","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9]]},"assertion":[{"value":"5 June 2023","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 September 2023","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Funding and\/or Conflicts of interests\/Competing interests"}},{"value":"The user study design has been approved for ethical correctness by one of the companies participated in the study as well as by the corresponding head of department at Fraunhofer IEM","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval"}},{"value":"For the user study, all 40 participants signed a written consent form in which they agreed to participate voluntarily in the study. They also agreed that the collected data can be used for research publication. The written consent form was obtained from all participants before the study","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to participate and publication"}}],"article-number":"118"}}