{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,15]],"date-time":"2025-11-15T02:24:36Z","timestamp":1763173476189,"version":"3.45.0"},"reference-count":48,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2025,7,21]],"date-time":"2025-07-21T00:00:00Z","timestamp":1753056000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,7,21]],"date-time":"2025-07-21T00:00:00Z","timestamp":1753056000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100023890","name":"Technische Universit\u00e4t Hamburg","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100023890","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Softw Syst Model"],"published-print":{"date-parts":[[2025,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Threat Modeling is an essential step in secure software system development. It is a (so far) manual, attacker-centric approach for identifying architecture-level security flaws during the planning phase of software systems. In recent years, academia has presented ideas to automate threat detection that do not focus on a particular class of security flaws but offer means of pattern-based security flaw descriptions. However, comparing presented ideas (tools) for automated threat detection contains the potential for unwilling bias or restricted information content. In this work, we investigate the process of comparing automatic security flaw detection tools, clarify common pitfalls during this process, and propose a fair, reproducible, and informative comparison approach to be used as a community standard. We additionally discuss the necessary steps for the community to effectively implement this approach and support improved comparisons and evaluations in the future. We use a previously published case study to determine problems with current comparison techniques and classify different levels of comparison to be used for future reference as our main contribution. As a consequence, we propose using a model-based approach for specifying security flaws and apply an existing natural language-based catalogue to this model-based approach. Furthermore, we introduce an inspection process model (for providing a standard to specify findings of a threat detection process) to streamline the evaluation and comparisons of automatic security flaw detection tools. We provide an exemplary evaluation of this detection guideline and inspection process model along the lines of both automatic approaches from the original case study. All artefacts of the work are publicly available to support the research community and to create a common baseline for future tool comparisons.<\/jats:p>","DOI":"10.1007\/s10270-025-01300-6","type":"journal-article","created":{"date-parts":[[2025,7,21]],"date-time":"2025-07-21T16:40:43Z","timestamp":1753116043000},"page":"1763-1796","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Automatic security-flaw detection - towards a fair evaluation and comparison"],"prefix":"10.1007","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6093-9229","authenticated-orcid":false,"given":"Bernhard J.","family":"Berger","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0392-6397","authenticated-orcid":false,"given":"Christina","family":"Plump","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2025,7,21]]},"reference":[{"key":"1300_CR1","doi-asserted-by":"crossref","unstructured":"McGraw, G.: Software security: building security in (Addison-Wesley Professional, 2006)","DOI":"10.1109\/ISSRE.2006.43"},{"key":"1300_CR2","volume-title":"Threat Modeling","author":"F Swiderski","year":"2004","unstructured":"Swiderski, F., Snyder, W.: Threat Modeling. Microsoft Press, USA (2004)"},{"key":"1300_CR3","doi-asserted-by":"publisher","unstructured":"Jiang, L., Chen, H., Deng, F.: A security evaluation method based on stride model for web service. In: 2nd international workshop on intelligent systems and applications, 1\u20135 (IEEE, 2010). (2010). https:\/\/doi.org\/10.1109\/IWISA.2010.5473445","DOI":"10.1109\/IWISA.2010.5473445"},{"key":"1300_CR4","doi-asserted-by":"crossref","unstructured":"Dhillon, D., Mishra, V.: Applied threat driven security verification. In: 2018 IEEE cybersecurity development (SecDev), 135\u2013135 (2018)","DOI":"10.1109\/SecDev.2018.00031"},{"key":"1300_CR5","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/1640162.1640171","volume":"34","author":"S Rehman","year":"2009","unstructured":"Rehman, S., Mustafa, K.: Research on software design level security vulnerabilities. SIGSOFT Softw. Eng. Notes 34, 1\u20135 (2009). https:\/\/doi.org\/10.1145\/1640162.1640171","journal-title":"SIGSOFT Softw. Eng. Notes"},{"key":"1300_CR6","volume-title":"Threat modeling \u2013 a practical guide for development teams","author":"I Tarandach","year":"2020","unstructured":"Tarandach, I., Coles, M.J.: Threat modeling \u2013 a practical guide for development teams, 1st edn. O\u2019Reilly Media Inc, USA (2020)","edition":"1"},{"key":"1300_CR7","unstructured":"Shostack, A.: Threat Modeling: Designing for Security 1st edn (Wiley Publishing, 2014)"},{"key":"1300_CR8","doi-asserted-by":"crossref","unstructured":"Yskout, K., Scandariato, R., Joosen, W.: Do security patterns really help designers?. In: Proceedings of the 37th international conference on software engineering - Volume 1, ICSE \u201915, 292\u2013302 (IEEE Press, 2015)","DOI":"10.1109\/ICSE.2015.49"},{"key":"1300_CR9","doi-asserted-by":"publisher","unstructured":"Tuma, K., Hosseini, D., Malamas, K., Scandariato, R.: Inspection guidelines to identify security design flaws. Proceedings of the 13th european conference on software architecture - Volume 2, ECSA \u201919, 116\u2013122 (Association for Computing Machinery, New York, NY, USA, 2019). https:\/\/doi.org\/10.1145\/3344948.3344995","DOI":"10.1145\/3344948.3344995"},{"key":"1300_CR10","doi-asserted-by":"publisher","unstructured":"Berger, B.\u00a0J., Plump, C.: Automatic security-flaw detection replication and comparison. In: ACM\/IEEE 26th international conference on model driven engineering languages and systems (MODELS), 84\u201394 (IEEE Press, 2023). (2023). https:\/\/doi.org\/10.1109\/MODELS58315.2023.00027","DOI":"10.1109\/MODELS58315.2023.00027"},{"key":"1300_CR11","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1109\/MSP.2011.47","volume":"9","author":"D Dhillon","year":"2011","unstructured":"Dhillon, D.: Developer-driven threat modeling: Lessons learned in the trenches. IEEE Secur. Privacy 9, 41\u201347 (2011)","journal-title":"IEEE Secur. Privacy"},{"key":"1300_CR12","unstructured":"MIL-STD-882C \u2013 System safety program requirements. In: Tech. Rep., Department of Defense (1993)"},{"key":"1300_CR13","unstructured":"Tarandach, I., Coles, M.\u00a0J.: Threat Modeling (O\u2019Reilly Media, Inc, 2020)"},{"key":"1300_CR14","doi-asserted-by":"publisher","DOI":"10.1002\/9781119653554","volume-title":"We have root: even more advice from Schneier on security","author":"B Schneier","year":"2019","unstructured":"Schneier, B.: We have root: even more advice from Schneier on security, 1st edn. John Wiley & Sons, Incorporated (2019)","edition":"1"},{"key":"1300_CR15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1155\/2014\/805856","volume":"2014","author":"M Frydman","year":"2014","unstructured":"Frydman, M., Ruiz, G., Heymann, E., C\u00e9sar, E., Miller, B.P.: Automating risk analysis of software design models. Sci. World J. 2014, 1\u201312 (2014). https:\/\/doi.org\/10.1155\/2014\/805856","journal-title":"Sci. World J."},{"key":"1300_CR16","doi-asserted-by":"publisher","unstructured":"Berger, B.\u00a0J., Sohr, K., Koschke, R., Caballero, J., Bodden, E., Athanasopoulos, E.: (eds) Automatically extracting threats from extended data flow diagrams. (eds Caballero, J., Bodden, E. & Athanasopoulos, E.) Engineering secure software and systems - 8th international symposium, ESSoS 2016, London, UK, April 6-8. Proceedings, Vol. 9639 of Lecture Notes in Computer Science, 56\u201371 (Springer, 2016). (2016). https:\/\/doi.org\/10.1007\/978-3-319-30806-7_4","DOI":"10.1007\/978-3-319-30806-7_4"},{"key":"1300_CR17","doi-asserted-by":"publisher","unstructured":"Tuma, K., Sion, L., Scandariato, R., Yskout, K.: Automating the early detection of security design flaws. Proceedings of the 23rd ACM\/IEEE international conference on model driven engineering languages and systems, MODELS \u201920, 332\u2013342 (Association for Computing Machinery, New York, NY, USA, 2020). https:\/\/doi.org\/10.1145\/3365438.3410954","DOI":"10.1145\/3365438.3410954"},{"key":"1300_CR18","doi-asserted-by":"publisher","first-page":"116514","DOI":"10.1109\/ACCESS.2022.3219063","volume":"10","author":"F De Rosa","year":"2022","unstructured":"De Rosa, F., Maunero, N., Prinetto, P., Talentino, F., Trussoni, M.: Threma: Ontology-based automated threat modeling for ict infrastructures. IEEE Access 10, 116514\u2013116526 (2022)","journal-title":"IEEE Access"},{"key":"1300_CR19","doi-asserted-by":"publisher","first-page":"111722","DOI":"10.1016\/j.jss.2023.111722","volume":"202","author":"S Schneider","year":"2023","unstructured":"Schneider, S., Scandariato, R.: Automatic extraction of security-rich dataflow diagrams for microservice applications written in java. J. Syst. Softw. 202, 111722 (2023). https:\/\/doi.org\/10.1016\/j.jss.2023.111722","journal-title":"J. Syst. Softw."},{"key":"1300_CR20","doi-asserted-by":"publisher","unstructured":"Granata, D., Rak, M., Salzillo, G.: Automated threat modeling approaches: comparison of open source tools, 250\u2013265 (Springer International Publishing, 2022). https:\/\/doi.org\/10.1007\/978-3-031-14179-9_17","DOI":"10.1007\/978-3-031-14179-9_17"},{"key":"1300_CR21","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-662-69306-3","author":"C Wohlin","year":"2024","unstructured":"Wohlin, C., et al.: Experimentation in software engineering (Springer. Berlin Heidelberg (2024). https:\/\/doi.org\/10.1007\/978-3-662-69306-3","journal-title":"Berlin Heidelberg"},{"key":"1300_CR22","doi-asserted-by":"crossref","unstructured":"Dennis, A.\u00a0R., Valacich, J.\u00a0S.: A replication manifesto. AIS Transactions on Replication Research, 1 (2015)","DOI":"10.17705\/1atrr.00001"},{"key":"1300_CR23","doi-asserted-by":"publisher","unstructured":"Berger, B.\u00a0J., Sohr, K., Koschke, R.: The architectural security tool suite - ARCHSEC. In: 19th international working conference on source code analysis and manipulation, SCAM 2019, Cleveland, OH, USA, September 30 - October 1, 250\u2013255 (IEEE, 2019). (2019). https:\/\/doi.org\/10.1109\/SCAM.2019.00035","DOI":"10.1109\/SCAM.2019.00035"},{"key":"1300_CR24","volume-title":"Structured analysis and system specification Yourdon computing series","author":"T DeMarco","year":"1979","unstructured":"DeMarco, T.: Structured analysis and system specification Yourdon computing series. Yourdon, Upper Saddle River, NJ (1979)"},{"key":"1300_CR25","unstructured":"Steinberg, D., Budinsky, F., Paternostro, M., Merks, E.: EMF: Eclipse modeling framework 2nd edn (Pearson International, 2008)"},{"key":"1300_CR26","doi-asserted-by":"publisher","unstructured":"Francis, N., et\u00a0al.: Cypher: An evolving query language for property graphs. Proceedings of the 2018 International Conference on Management of Data, SIGMOD \u201918, 1433\u20131445 (Association for Computing Machinery, New York, NY, USA, 2018). https:\/\/doi.org\/10.1145\/3183713.3190657","DOI":"10.1145\/3183713.3190657"},{"key":"1300_CR27","unstructured":"Berger, B.\u00a0J., Sohr, K., Kalinna, U.\u00a0H.: in Architekturelle Sicherheitsanalyse f\u00fcr Android (ed.Horster, P.) D $$\\bullet $$ A$$\\bullet $$CH Security 2014: Bestandsaufnahme - Konzepte - Anwendungen - Perspektiven 287\u2013298 (Peter Schartner and Peter Lipp, 2014)"},{"key":"1300_CR28","doi-asserted-by":"publisher","first-page":"214","DOI":"10.1016\/j.scico.2007.05.004","volume":"68","author":"D Varr\u00f3","year":"2007","unstructured":"Varr\u00f3, D., Balogh, A.: The model transformation language of the viatra2 framework. Sci. Comput. Program. 68, 214\u2013234 (2007)","journal-title":"Sci. Comput. Program."},{"key":"1300_CR29","doi-asserted-by":"publisher","first-page":"275","DOI":"10.1016\/j.jss.2018.06.073","volume":"144","author":"K Tuma","year":"2018","unstructured":"Tuma, K., Calikli, G., Scandariato, R.: Threat analysis of software systems: a systematic literature review. J. Syst. Softw. 144, 275\u2013294 (2018)","journal-title":"J. Syst. Softw."},{"key":"1300_CR30","doi-asserted-by":"publisher","unstructured":"J\u00fcrjens, J., Hu\u00dfmann, H.: Towards development of secure systems using umlsec. (ed.Hu\u00dfmann, H.) Fundamental Approaches to Software Engineering, 4th International Conference, FASE 2001 Held as Part of the Joint European conferences on theory and practice of software, ETAPS 2001 Genova, Italy, April 2-6, Proceedings, Vol. 2029 of Lecture Notes in Computer Science, 187\u2013200 (Springer, 2001). (2001). https:\/\/doi.org\/10.1007\/3-540-45314-8_14","DOI":"10.1007\/3-540-45314-8_14"},{"key":"1300_CR31","doi-asserted-by":"crossref","unstructured":"J\u00fcrjens, J., J\u00e9z\u00e9quel, J.-M., Hussmann, H., Cook, S.: (eds) Umlsec: Extending uml for secure systems development. (eds J\u00e9z\u00e9quel, J.-M., Hussmann, H. & Cook, S.) $$<<$$UML$$>>$$ 2002 \u2014 The Unified Modeling Language, 412\u2013425 (Springer Berlin Heidelberg, Berlin, Heidelberg, 2002)","DOI":"10.1007\/3-540-45800-X_32"},{"key":"1300_CR32","doi-asserted-by":"publisher","unstructured":"J\u00fcrjens, J., Houmb, S.\u00a0H., Reis, R.: (ed.) Risk-driven development of security-critical systems using umlsec. (ed.Reis, R.) Information Technology, Selected Tutorials, IFIP 18th world computer congress, tutorials, 22-27 August, Toulouse, France, Vol. 157 of IFIP, 21\u201353 (Kluwer\/Springer, 2004). (2004). https:\/\/doi.org\/10.1007\/1-4020-8159-6_2","DOI":"10.1007\/1-4020-8159-6_2"},{"key":"1300_CR33","doi-asserted-by":"crossref","unstructured":"Best, B., J\u00fcrjens, J., Nuseibeh, B.: Model-based security engineering of distributed information systems using umlsec. In: 29th international conference on software engineering (ICSE\u201907), 581\u2013590 (2007)","DOI":"10.1109\/ICSE.2007.55"},{"key":"1300_CR34","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1016\/j.entcs.2008.11.008","volume":"220","author":"J J\u00fcrjens","year":"2008","unstructured":"J\u00fcrjens, J.: Model-based security testing using UMLSEC: a case study. Electron. Notes Theoretical Comput. Sci. 220, 93\u2013104 (2008)","journal-title":"Electron. Notes Theoretical Comput. Sci."},{"key":"1300_CR35","doi-asserted-by":"publisher","unstructured":"Ruhroth, T., J\u00fcrjens, J.: Supporting security assurance in the context of evolution: Modular modeling and analysis with umlsec. 14th International IEEE symposium on high-assurance systems engineering, HASE, Omaha, NE, USA, October 25-27, 2012, 177\u2013184 (IEEE Computer Society, 2012). (2012). https:\/\/doi.org\/10.1109\/HASE.2012.35","DOI":"10.1109\/HASE.2012.35"},{"key":"1300_CR36","doi-asserted-by":"publisher","unstructured":"Abi-Antoun, M., Wang, D., Torr, P.: Checking threat modeling data flow diagrams for implementation conformance and security. Proceedings of the Twenty-Second IEEE\/ACM international conference on automated software engineering, ASE \u201907, 393\u2013396 (Association for Computing Machinery, New York, NY, USA, 2007). https:\/\/doi.org\/10.1145\/1321631.1321692","DOI":"10.1145\/1321631.1321692"},{"key":"1300_CR37","doi-asserted-by":"publisher","unstructured":"Abi-Antoun, M., Barnes, J.\u00a0M.: Analyzing security architectures. Proceedings of the IEEE\/ACM international conference on automated software engineering, ASE \u201910, 3\u201312 (Association for Computing Machinery, New York, NY, USA, 2010). https:\/\/doi.org\/10.1145\/1858996.1859001","DOI":"10.1145\/1858996.1859001"},{"key":"1300_CR38","doi-asserted-by":"crossref","unstructured":"Vanciu, R., Abi-Antoun, M.: Ownership object graphs with dataflow edges. 19th Working conference on reverse engineering, 267\u2013276 (2012)","DOI":"10.1109\/WCRE.2012.36"},{"key":"1300_CR39","unstructured":"Garlan, D. et\u00a0al.: Documenting software architectures: views and beyond 2nd edn (Addison-Wesley Professional, 2010)"},{"key":"1300_CR40","doi-asserted-by":"publisher","first-page":"364","DOI":"10.1109\/32.917525","volume":"27","author":"GC Murphy","year":"2001","unstructured":"Murphy, G.C., Notkin, D., Sullivan, K.J.: Software reflexion models: Bridging the gap between design and implementation. IEEE Trans. Softw. Eng. 27, 364\u2013380 (2001). https:\/\/doi.org\/10.1109\/32.917525","journal-title":"IEEE Trans. Softw. Eng."},{"key":"1300_CR41","doi-asserted-by":"crossref","unstructured":"Almorsy, M., Grundy, J., Ibrahim, A.\u00a0S.: Supporting automated vulnerability analysis using formalized vulnerability signatures. 2012 Proceedings of the 27th IEEE\/ACM international conference on automated software engineering, 100\u2013109 (2012)","DOI":"10.1145\/2351676.2351691"},{"key":"1300_CR42","doi-asserted-by":"crossref","unstructured":"Almorsy, M., Grundy, J., Ibrahim, A.\u00a0S.: Automated software architecture security risk analysis using formalized signatures. Proceedings of the 2013 international conference on software engineering, ICSE \u201913, 662\u2013671 (IEEE Press, 2013)","DOI":"10.1109\/ICSE.2013.6606612"},{"key":"1300_CR43","doi-asserted-by":"publisher","DOI":"10.1145\/3699711","author":"N Shiri Harzevili","year":"2024","unstructured":"Shiri Harzevili, N., et al.: A systematic literature review on automated software vulnerability detection using machine learning. ACM Comput. Surv. (2024). https:\/\/doi.org\/10.1145\/3699711","journal-title":"ACM Comput. Surv."},{"key":"1300_CR44","unstructured":"William G., Cochran, G. M.\u00a0C.: Experimental Designs 2. edn (Wiley, 1994)"},{"key":"1300_CR45","doi-asserted-by":"crossref","unstructured":"Beyer, D., Strej\u010dek, J., Gurfinkel, A., Heule, M.: (eds) Improvements in software verification and witness validation: Sv-comp. (eds Gurfinkel, A. & Heule, M.) Tools and Algorithms for the construction and analysis of systems, 151\u2013186 (Springer Nature Switzerland, Cham, 2025) (2025)","DOI":"10.1007\/978-3-031-90660-2_9"},{"key":"1300_CR46","doi-asserted-by":"publisher","unstructured":"Ab.\u00a0Rahim, L., Whittle, J.: A survey of approaches for verifying model transformations. Software & Systems Modeling 14, 1003\u2013s1028 (2013). https:\/\/doi.org\/10.1007\/s10270-013-0358-0","DOI":"10.1007\/s10270-013-0358-0"},{"key":"1300_CR47","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1145\/185403.185412","volume":"26","author":"CE Landwehr","year":"1994","unstructured":"Landwehr, C.E., Bull, A.R., McDermott, J.P., Choi, W.S.: A taxonomy of computer program security flaws. ACM Comput. Surveys 26, 211\u2013254 (1994). https:\/\/doi.org\/10.1145\/185403.185412","journal-title":"ACM Comput. Surveys"},{"key":"1300_CR48","doi-asserted-by":"publisher","unstructured":"Santos, J. C.\u00a0S., Tarrit, K., Mirakhorli, M., Malavolta, I., Capilla, R.: (eds) A catalog of security architecture weaknesses. (eds Malavolta, I. & Capilla, R.) 2017 IEEE international conference on software architecture workshops (ICSAW), 220\u2013223 (IEEE, 2017). https:\/\/doi.org\/10.1109\/ICSAW.2017.25","DOI":"10.1109\/ICSAW.2017.25"}],"container-title":["Software and Systems Modeling"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10270-025-01300-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10270-025-01300-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10270-025-01300-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,15]],"date-time":"2025-11-15T02:20:01Z","timestamp":1763173201000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10270-025-01300-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,21]]},"references-count":48,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2025,12]]}},"alternative-id":["1300"],"URL":"https:\/\/doi.org\/10.1007\/s10270-025-01300-6","relation":{},"ISSN":["1619-1366","1619-1374"],"issn-type":[{"type":"print","value":"1619-1366"},{"type":"electronic","value":"1619-1374"}],"subject":[],"published":{"date-parts":[[2025,7,21]]},"assertion":[{"value":"5 May 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 May 2025","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 May 2025","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 July 2025","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}