{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,23]],"date-time":"2025-12-23T05:04:05Z","timestamp":1766466245054,"version":"build-2065373602"},"reference-count":219,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2022,9,24]],"date-time":"2022-09-24T00:00:00Z","timestamp":1663977600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"MIUR-PRIN 2020TL3X8X project T-LADIES (Typeful Language Adaptation for Dynamic, Interacting and Evolving Systems)"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Software"],"abstract":"<jats:p>Empirical studies on formal methods and tools are rare. In this paper, we provide guidelines for such studies. We mention their main ingredients and then define nine different study strategies (usability testing, laboratory experiments with software and human subjects, case studies, qualitative studies, surveys, judgement studies, systematic literature reviews, and systematic mapping studies) and discuss for each of them their crucial characteristics, the difficulties of applying them to formal methods and tools, typical threats to validity, their maturity in formal methods, pointers to external guidelines, and pointers to studies in other fields. We conclude with a number of challenges for empirical formal methods.<\/jats:p>","DOI":"10.3390\/software1040017","type":"journal-article","created":{"date-parts":[[2022,9,25]],"date-time":"2022-09-25T21:14:28Z","timestamp":1664140468000},"page":"381-416","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Empirical Formal Methods: Guidelines for Performing Empirical Studies on Formal Methods"],"prefix":"10.3390","volume":"1","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2930-6367","authenticated-orcid":false,"given":"Maurice H.","family":"ter Beek","sequence":"first","affiliation":[{"name":"Formal Methods and Tools Lab, ISTI\u2013CNR, Via G. Moruzzi 1, 56124 Pisa, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0636-5663","authenticated-orcid":false,"given":"Alessio","family":"Ferrari","sequence":"additional","affiliation":[{"name":"Formal Methods and Tools Lab, ISTI\u2013CNR, Via G. Moruzzi 1, 56124 Pisa, Italy"}]}],"member":"1968","published-online":{"date-parts":[[2022,9,24]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"876","DOI":"10.1007\/s11390-018-1864-x","article-title":"Empirical Research in Software Engineering\u2014A Literature Survey","volume":"33","author":"Zhang","year":"2018","journal-title":"J. Comput. Sci. Technol."},{"key":"ref_2","first-page":"10","article-title":"Status of Empirical Research in Software Engineering","volume":"Volume 4336","author":"Basili","year":"2006","journal-title":"Proceedings of the International Workshop on Empirical Software Engineering Issues: Critical Assessment and Future Directions"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1016\/j.infsof.2014.11.005","article-title":"An empirical research agenda for understanding formal methods productivity","volume":"60","author":"Jeffery","year":"2015","journal-title":"Inf. Softw. Technol."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Gleirscher, M., van de Pol, J., and Woodcock, J. (2022, September 22). A Manifesto for Applicable Formal Methods. Available online: https:\/\/arxiv.org\/abs\/2112.12758.","DOI":"10.1007\/s10270-023-01124-2"},{"key":"ref_5","unstructured":"Huisman, M., Gurov, D., and Malkis, A. (2022, September 22). Formal Methods: From Academia to Industrial Practice. A Travel Guide. Available online: https:\/\/arxiv.org\/abs\/2002.07279."},{"key":"ref_6","unstructured":"Ferrari, A., Mazzanti, F., Basile, D., ter Beek, M.H., and Fantechi, A. (July, January 27). Comparing Formal Tools for System Design: A Judgment Study. Proceedings of the 42nd ACM International Conference on Software Engineering (ICSE\u201920), Seoul, Korea."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Wohlin, C., Runeson, P., H\u00f6st, M., Ohlsson, M.C., Regnell, B., and Wessl\u00e9n, A. (2012). Experimentation in Software Engineering, Springer.","DOI":"10.1007\/978-3-642-29044-2"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Shull, F., Singer, J., and Sj\u00f8berg, D.I.K. (2008). Guide to Advanced Empirical Software Engineering, Springer.","DOI":"10.1007\/978-1-84800-044-5"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Felderer, M., and Travassos, G.H. (2020). Contemporary Empirical Methods in Software Engineering, Springer.","DOI":"10.1007\/978-3-030-32489-6"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"721","DOI":"10.1109\/TSE.2002.1027796","article-title":"Preliminary Guidelines for Empirical Research in Software Engineering","volume":"28","author":"Kitchenham","year":"2002","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Shull, F., Singer, J., and Sj\u00f8berg, D.I.K. (2008). Selecting Empirical Methods for Software Engineering Research. Guide to Advanced Empirical Software Engineering, Springer.","DOI":"10.1007\/978-1-84800-044-5"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3241743","article-title":"The ABC of software engineering research","volume":"27","author":"Stol","year":"2018","journal-title":"ACM Trans. Softw. Eng. Methodol."},{"key":"ref_13","unstructured":"Ralph, P. (2022, September 22). Empirical Standards for Software Engineering Research. Available online: https:\/\/arxiv.org\/abs\/2010.03525."},{"key":"ref_14","unstructured":"Box, G.E.P., Hunter, J.S., and Hunter, W.G. (2005). Statistics for Experimenters: Design, Innovation, and Discovery, Wiley."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1049\/cce:19970304","article-title":"DESMET: A methodology for evaluating software engineering methods and tools","volume":"8","author":"Kitchenham","year":"1997","journal-title":"Comput. Control Eng. J."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s10009-017-0469-y","article-title":"Reliable benchmarking: Requirements and solutions","volume":"21","author":"Beyer","year":"2019","journal-title":"Int. J. Softw. Tools Technol. Transf."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Kordon, F., and Hulin-Hubard, F. (2014, January 23\u201327). BenchKit, a Tool for Massive Concurrent Benchmarking. Proceedings of the 14th International Conference on Application of Concurrency to System Design (ACSD\u201914), IEEE, Tunis La Marsa, Tunisia.","DOI":"10.1109\/ACSD.2014.12"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1411","DOI":"10.1002\/spe.2382","article-title":"DataMill: A distributed heterogeneous infrastructure for robust experimentation","volume":"46","author":"Petkovich","year":"2016","journal-title":"Softw. Pract. Exp."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"367","DOI":"10.1007\/978-3-319-08587-6_28","article-title":"StarExec: A Cross-Community Infrastructure for Logic Solving","volume":"Volume 8562","author":"Demri","year":"2014","journal-title":"Proceedings of the 7th International Joint Conference on Automated Reasoning (IJCAR\u201914)"},{"key":"ref_20","first-page":"89","article-title":"The International SAT Solver Competitions","volume":"33","author":"Berre","year":"2012","journal-title":"AI Mag."},{"key":"ref_21","first-page":"99","article-title":"The CADE ATP System Competition-CASC","volume":"37","author":"Sutcliffe","year":"2016","journal-title":"AI Mag."},{"key":"ref_22","first-page":"3","article-title":"TOOLympics 2019: An Overview of Competitions in Formal Methods","volume":"Volume 11429","author":"Beyer","year":"2019","journal-title":"Proceedings of the 25th International Conference on Tools and Algorithms for the Construction and Analysis of Systems: TOOLympics (TACAS\u201919)"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1145\/2464526.2464530","article-title":"Artifact Evaluation for Software Conferences","volume":"38","author":"Krishnamurthi","year":"2013","journal-title":"ACM Sigsoft Softw. Eng. Notes"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1145\/2658987","article-title":"The Real Software Crisis: Repeatability as a Core Value\u2013Sharing experiences running artifact evaluation committees for five major conferences","volume":"58","author":"Krishnamurthi","year":"2015","journal-title":"Commun. ACM"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1007\/978-3-030-58298-2_1","article-title":"The 2020 Expert Survey on Formal Methods","volume":"Volume 12327","year":"2020","journal-title":"Proceedings of the 25th International Conference on Formal Methods for Industrial Critical Systems (FMICS\u201920)"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1145\/2442776.2442781","article-title":"R3\u2013Repeatability, Reproducibility and Rigor","volume":"47","author":"Vitek","year":"2012","journal-title":"ACM Sigplan Not."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"110825","DOI":"10.1016\/j.jss.2020.110825","article-title":"A comprehensive study of automatic program repair on the QuixBugs benchmark","volume":"171","author":"Ye","year":"2021","journal-title":"J. Syst. Softw."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Su, T., Wang, J., and Su, Z. (2021, January 23\u201328). Benchmarking automated GUI testing for Android against real-world bugs. Proceedings of the 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE\u201921), Athens, Greece.","DOI":"10.1145\/3468264.3468620"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"811","DOI":"10.1109\/TSE.2017.2724538","article-title":"A Comparative Study to Benchmark Cross-Project Defect Prediction Approaches","volume":"44","author":"Herbold","year":"2018","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1109\/TSE.2011.122","article-title":"Empirical Principles and an Industrial Case Study in Retrieving Equivalent Requirements via Natural Language Processing Techniques","volume":"39","author":"Falessi","year":"2013","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1007\/s00766-016-0251-9","article-title":"On the automatic classification of app reviews","volume":"21","author":"Maalej","year":"2016","journal-title":"Requir. Eng."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"5454","DOI":"10.1007\/s10664-020-09864-1","article-title":"Automated demarcation of requirements in textual specifications: A machine learning-based approach","volume":"25","author":"Abualhaija","year":"2020","journal-title":"Empir. Softw. Eng."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s10009-016-0446-x","article-title":"The Physics of Software Tools: SWOT Analysis and Vision","volume":"19","author":"Steffen","year":"2017","journal-title":"Int. J. Softw. Tools Technol. Transfer"},{"key":"ref_34","first-page":"186","article-title":"Reflections on Bernhard Steffen\u2019s Physics of Software Tools","volume":"Volume 11200","author":"Margaria","year":"2019","journal-title":"Models, Mindsets, Meta: The What, the How, and the Why Not?"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"130","DOI":"10.1145\/1735223.1735255","article-title":"Number of people required for usability evaluation: The 10 \u00b1 2 rule","volume":"53","author":"Hwang","year":"2010","journal-title":"Commun. ACM"},{"key":"ref_36","first-page":"34","article-title":"How To Specify the Participant Group Size for Usability Studies: A Practitioner\u2019s Guide","volume":"5","author":"Macefield","year":"2009","journal-title":"J. Usability Stud."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"308","DOI":"10.1109\/32.991322","article-title":"Formal Methods Application: An Empirical Tale of Software Development","volume":"28","author":"Sobel","year":"2002","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"567","DOI":"10.1109\/TSE.2003.1205183","article-title":"Comments on \u201cFormal Methods Application: An Empirical Tale of Software Development\u201d","volume":"29","author":"Berry","year":"2003","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"572","DOI":"10.1109\/TSE.2003.1205184","article-title":"Response to \u201cComments on \u2018Formal Methods Application: An Empirical Tale of Software Development\u2019\u201d","volume":"29","author":"Sobel","year":"2003","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1109\/2.566148","article-title":"Investigating the Influence of Formal Methods","volume":"30","author":"Pfleeger","year":"1997","journal-title":"IEEE Comput."},{"key":"ref_41","unstructured":"Cook, C.R., Scholtz, J.C., and Spohrer, J.C. (1993, January 3\u20135). Comparing the Comprehensibility of Textual and Graphical Programs: The Case of Petri Nets. Proceedings of the 5th Workshop on Empirical Studies of Programmers (ESP\u201993), Palo Alto, CA, USA."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/S0164-1212(98)00003-X","article-title":"Measuring the comprehensibility of Z specifications","volume":"42","author":"Finney","year":"1998","journal-title":"J. Syst. Softw."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"955","DOI":"10.1016\/j.infsof.2004.04.003","article-title":"Experimental comparison of the comprehensibility of a Z specification and its implementation in Java","volume":"46","author":"Snook","year":"2004","journal-title":"Inf. Softw. Technol."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1006\/jvlc.2001.0213","article-title":"An Experiment to Compare the Comprehensibility of Textual and Visual Forms of Algebraic Specifications","volume":"13","author":"Neary","year":"2002","journal-title":"J. Vis. Lang. Comput."},{"key":"ref_45","unstructured":"Carew, D., Exton, C., and Buckley, J. (2005, January 17\u201318). An empirical investigation of the comprehensibility of requirements specifications. Proceedings of the International Symposium on Empirical Software Engineering (ISESE\u201905), IEEE, Noosa Heads, Australia."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Kitchenham, B.A., Brereton, P., and Turner, M. (2007, January 2\u20133). Experimental Comparison of the Comprehensibility of a UML-based Formal Specification versus a Textual One. Proceedings of the 11th International Conference on Evaluation and Assessment in Software Engineering (EASE\u201907), Keele, UK.","DOI":"10.14236\/ewic\/EASE2007.0"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Razali, R., Snook, C.F., and Poppleton, M.R. (2007, January 5). Comprehensibility of UML-Based Formal Model: A Series of Controlled Experiments. Proceedings of the 1st ACM International Workshop on Empirical Assessment of Software Engineering Languages and Technologies (WEASELTech\u201907), Atlanta, GA, USA.","DOI":"10.1145\/1353673.1353680"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Zimmerman, M.K., Lundqvist, K., and Leveson, N.G. (2002, January 19\u201325). Investigating the Readability of State-Based Formal Requirements Specification Languages. Proceedings of the 24th ACM International Conference on Software Engineering (ICSE\u201902), Orlando, FL, USA.","DOI":"10.1145\/581344.581347"},{"key":"ref_49","first-page":"434","article-title":"Comparing the Control-Flow of EPC and Petri Net from the End-User Perspective","volume":"Volume 3649","author":"Benatallah","year":"2005","journal-title":"Proceedings of the 3rd International Conference on Business Process Management (BPM\u201905)"},{"key":"ref_50","first-page":"48","article-title":"What Makes Process Models Understandable?","volume":"Volume 4714","author":"Alonso","year":"2007","journal-title":"Proceedings of the 5th International Conference on Business Process Management (BPM\u201907)"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"449","DOI":"10.1109\/TSMCA.2010.2087017","article-title":"A Study Into the Factors That Influence the Understandability of Business Process Models","volume":"41","author":"Reijers","year":"2011","journal-title":"IEEE Trans. Syst. Man Cybern. Part A"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"501","DOI":"10.1007\/978-3-319-11653-2_31","article-title":"Comprehending Feature Models Expressed in CVL","volume":"Volume 8767","author":"Dingel","year":"2014","journal-title":"Proceedings of the 17th International Conference on Model-Driven Engineering Languages and Systems (MoDELS\u201914)"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"678","DOI":"10.1007\/s10664-012-9234-8","article-title":"Comprehensibility of UML-based software product line specifications\u2014A controlled experiment","volume":"19","author":"Sturm","year":"2014","journal-title":"Empir. Softw. Eng."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Labunets, K., Massacci, F., Paci, F., and Tran, L.M.S. (2013, January 10\u201311). An Experimental Comparison of Two Risk-Based Security Methods. Proceedings of the IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM\u201913), Baltimore, MD, USA.","DOI":"10.1109\/ESEM.2013.29"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Labunets, K., Paci, F., Massacci, F., and Ruprai, R.S. (2014, January 25). An Experiment on Comparing Textual vs. Visual Industrial Methods for Security Risk Assessment. Proceedings of the 4th IEEE International Workshop on Empirical Requirements Engineering (EmpiRE\u201914), Karlskrona, Sweden.","DOI":"10.1109\/EmpiRE.2014.6890113"},{"key":"ref_56","first-page":"191","article-title":"On the Equivalence Between Graphical and Tabular Representations for Security Risk Assessment","volume":"Volume 10153","author":"Perini","year":"2017","journal-title":"Proceedings of the 23rd International Working Conference on Requirements Engineering: Foundation for Software Quality (REFSQ\u201917)"},{"key":"ref_57","first-page":"21","article-title":"Attack Trees","volume":"24","author":"Schneier","year":"1999","journal-title":"Dr. Dobb\u2019s J. Softw. Tools"},{"key":"ref_58","first-page":"186","article-title":"Foundations of Attack Trees","volume":"Volume 3935","author":"Won","year":"2005","journal-title":"Proceedings of the 8th International Conference on Information Security and Cryptology (ICISC\u201905)"},{"key":"ref_59","first-page":"80","article-title":"Foundations of Attack-Defense Trees","volume":"Volume 6561","author":"Degano","year":"2010","journal-title":"Proceedings of the 7th International Workshop on Formal Aspects of Security and Trust (FAST\u201910)"},{"key":"ref_60","unstructured":"Amenaza Technologies Limited (2022, September 22). The SecuITree\u00ae BurgleHouse Tutorial (a.k.a., Who wants to be a Cat Burglar?). Available online: https:\/\/www.amenaza.com\/downloads\/docs\/Tutorial.pdf."},{"key":"ref_61","first-page":"115","article-title":"New Directions in Attack Tree Research: Catching up with Industrial Needs","volume":"Volume 10744","author":"Liu","year":"2017","journal-title":"Proceedings of the 4th International Workshop on Graphical Models for Security (GraMSec\u201917)"},{"key":"ref_62","first-page":"214","article-title":"Assessing Security of Cryptocurrencies with Attack-Defense Trees: Proof of Concept and Future Directions","volume":"Volume 12819","author":"Cerone","year":"2021","journal-title":"Proceedings of the 18th International Colloquium on Theoretical Aspects of Computing (ICTAC\u201921)"},{"key":"ref_63","doi-asserted-by":"crossref","first-page":"110","DOI":"10.1007\/s10664-013-9279-3","article-title":"A practical guide to controlled experiments of software engineering tools with human participants","volume":"20","author":"Ko","year":"2015","journal-title":"Empir. Softw. Eng."},{"key":"ref_64","first-page":"1742","article-title":"A Procedure and Guidelines for Analyzing Groups of Software Engineering Replications","volume":"47","author":"Santos","year":"2019","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_65","unstructured":"Trochim, W.M.K. (2022, September 20). The Research Methods Knowledge Base. Available online: https:\/\/conjointly.com\/kb\/."},{"key":"ref_66","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3469888","article-title":"Psychometrics in Behavioral Software Engineering: A Methodological Introduction with Guidelines","volume":"31","author":"Graziotin","year":"2021","journal-title":"ACM Trans. Softw. Eng. Methodol."},{"key":"ref_67","unstructured":"Motulsky, H. (2013). Intuitive Biostatistics: A Nonmathematical Guide to Statistical Thinking, Oxford University Press."},{"key":"ref_68","first-page":"1786","article-title":"Bayesian Data Analysis in Empirical Software Engineering Research","volume":"47","author":"Furia","year":"2019","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_69","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1109\/TSE.2012.27","article-title":"Assessing the Effectiveness of Sequence Diagrams in the Comprehension of Functional Requirements: Results from a Family of Five Experiments","volume":"39","author":"Gravino","year":"2013","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_70","doi-asserted-by":"crossref","first-page":"3356","DOI":"10.1016\/j.ins.2011.04.005","article-title":"Evaluating requirements modeling methods based on user perceptions: A family of experiments","volume":"181","author":"Genero","year":"2011","journal-title":"Inf. Sci."},{"key":"ref_71","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s10664-020-09895-8","article-title":"A family of experiments on test-driven development","volume":"26","author":"Santos","year":"2021","journal-title":"Empir. Softw. Eng."},{"key":"ref_72","doi-asserted-by":"crossref","first-page":"936","DOI":"10.1109\/TSE.2019.2909033","article-title":"Requirements Framing Affects Design Creativity","volume":"47","author":"Mohanani","year":"2021","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_73","doi-asserted-by":"crossref","first-page":"427","DOI":"10.1109\/TSE.2015.2494588","article-title":"Effect of Domain Knowledge on Elicitation Effectiveness: An Internally Replicated Controlled Experiment","volume":"42","author":"Aranda","year":"2016","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_74","doi-asserted-by":"crossref","first-page":"106337","DOI":"10.1016\/j.infsof.2020.106337","article-title":"Comparing manual and automated feature location in conceptual models: A Controlled experiment","volume":"125","author":"Cetina","year":"2020","journal-title":"Inf. Softw. Technol."},{"key":"ref_75","unstructured":"Dumas, J.S., and Redish, J. (1999). A Practical Guide to Usability Testing, Intellect."},{"key":"ref_76","doi-asserted-by":"crossref","unstructured":"Nielsen, J., and Molich, R. (1990, January 1\u20135). Heuristic evaluation of user interfaces. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI\u201990), Seattle, WA, USA.","DOI":"10.1145\/97243.97281"},{"key":"ref_77","doi-asserted-by":"crossref","first-page":"741","DOI":"10.1080\/10447311003781409","article-title":"State of the Art on the Cognitive Walkthrough Method, Its Variants and Evolutions","volume":"26","author":"Mahatody","year":"2010","journal-title":"Int. J. Hum. Comput. Interact."},{"key":"ref_78","unstructured":"(2018). Ergonomics of Human-System Interaction\u2013Part 11: Usability: Definitions and Concepts (Standard No. ISO 9241-11:2018)."},{"key":"ref_79","unstructured":"Albers, M.J., and Mazur, M.B. (2003). The Five Dimensions of Usability. Content and Complexity: Information Design in Technical Communication, Taylor & Francis. Chapter 4."},{"key":"ref_80","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1145\/1039539.1039541","article-title":"Usability engineering methods for software developers","volume":"48","author":"Holzinger","year":"2005","journal-title":"Commun. ACM"},{"key":"ref_81","doi-asserted-by":"crossref","unstructured":"Sauro, J., and Lewis, J.R. (2016). Standardized usability questionnaires. Quantifying the User Experience: Practical Statistics for User Research, Morgan Kaufmann. Chapter 8.","DOI":"10.1016\/B978-0-12-802308-2.00008-4"},{"key":"ref_82","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1016\/j.ijhcs.2005.06.002","article-title":"Current practice in measuring usability: Challenges to usability studies and research","volume":"64","year":"2006","journal-title":"Int. J. Hum. Comput. Stud."},{"key":"ref_83","doi-asserted-by":"crossref","first-page":"319","DOI":"10.2307\/249008","article-title":"Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology","volume":"13","author":"Davis","year":"1989","journal-title":"MIS Q."},{"key":"ref_84","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1007\/s10799-013-0162-0","article-title":"Exploring the relationship between technology acceptance model and usability test","volume":"14","author":"Lin","year":"2013","journal-title":"Inf. Technol. Manag."},{"key":"ref_85","doi-asserted-by":"crossref","unstructured":"Ferrari, A., Mazzanti, F., Basile, D., and ter Beek, M.H. (2021). Systematic Evaluation and Usability Analysis of Formal Methods Tools for Railway Signaling System Design. IEEE Trans. Softw. Eng.","DOI":"10.1109\/TSE.2021.3124677"},{"key":"ref_86","doi-asserted-by":"crossref","unstructured":"Kelley, J.F. (1983, January 12\u201315). An empirical methodology for writing user-friendly natural language computer applications. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201983), ACM, Boston, MA, USA.","DOI":"10.1145\/800045.801609"},{"key":"ref_87","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1037\/h0046016","article-title":"Convergent and discriminant validation by the multitrait-multimethod matrix","volume":"56","author":"Campbell","year":"1959","journal-title":"Psychol. Bull."},{"key":"ref_88","unstructured":"Kadoda, G.F. (1997). Formal Software Development Tools (An Investigation into Usability). [Ph.D. Thesis, Loughborough University]."},{"key":"ref_89","unstructured":"Shackel, B. (1986, January 23\u201326). Ergonomics in Design for Usability. Proceedings of the 2nd Conference of the British Computer Society, Human Computer Interaction Specialist Group on People and Computers: Designing for Usability, York, UK."},{"key":"ref_90","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1006\/jvlc.1996.0009","article-title":"Usability Analysis of Visual Programming Environments: A \u2018Cognitive Dimensions\u2019 Framework","volume":"7","author":"Green","year":"1996","journal-title":"J. Vis. Lang. Comput."},{"key":"ref_91","unstructured":"Hussey, A., MacColl, I., and Carrington, D.A. (2001, January 27\u201328). Assessing Usability from Formal User-Interface Designs. Proceedings of the 13th IEEE Australian Software Engineering Conference (ASWEC\u201901), Canberra, Australia."},{"key":"ref_92","unstructured":"Chechik, M. (December, January 30). SC(R)3: Towards Usability of Formal Methods. Proceedings of the 8th Conference of the Centre for Advanced Studies on Collaborative Research (CASCON\u201998), IBM, Toronto, ON, Canada."},{"key":"ref_93","doi-asserted-by":"crossref","unstructured":"Loer, K., and Harrison, M.D. (2002, January 23\u201327). Towards Usable and Relevant Model Checking Techniques for the Analysis of Dependable Interactive Systems. Proceedings of the 17th International Conference on Automated Software Engineering (ASE\u201902), IEEE, Edinburgh, UK.","DOI":"10.1109\/ASE.2002.1115016"},{"key":"ref_94","doi-asserted-by":"crossref","first-page":"469","DOI":"10.1007\/s10515-006-7999-y","article-title":"An integrated framework for the analysis of dependable interactive systems (IFADIS): Its tool support and evaluation","volume":"13","author":"Loer","year":"2006","journal-title":"Autom. Softw. Eng."},{"key":"ref_95","first-page":"1","article-title":"Advances in Usability of Formal Methods for Code Verification with Frama-C","volume":"77","author":"Maroneze","year":"2019","journal-title":"Electron. Commun. Eur. Assoc. Softw. Sci. Technol."},{"key":"ref_96","first-page":"61","article-title":"Addressing Usability in a Formal Development Environment","volume":"Volume 12232","author":"Sekerinski","year":"2019","journal-title":"Revised Selected Papers of the International Workshops at the 3rd World Congress on Formal Methods (FM\u201919)"},{"key":"ref_97","unstructured":"Rubin, J., and Chisnell, D. (2008). Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests, Wiley."},{"key":"ref_98","doi-asserted-by":"crossref","unstructured":"Sagar, K., and Saha, A. (2017). A systematic review of software usability studies. Int. J. Inf. Technol., 1\u201324.","DOI":"10.1007\/s41870-017-0048-1"},{"key":"ref_99","doi-asserted-by":"crossref","first-page":"103363","DOI":"10.1016\/j.csi.2019.103363","article-title":"How are UML class diagrams built in practice? A usability study of two UML tools: Magicdraw and Papyrus","volume":"67","author":"Planas","year":"2020","journal-title":"Comput. Stand. Interfaces"},{"key":"ref_100","doi-asserted-by":"crossref","first-page":"106171","DOI":"10.1016\/j.infsof.2019.08.003","article-title":"Assessing the effectiveness of goal-oriented modeling languages: A family of experiments","volume":"116","year":"2019","journal-title":"Inf. Softw. Technol."},{"key":"ref_101","doi-asserted-by":"crossref","unstructured":"Ryan, T.P. (2013). Sample Size Determination and Power, Wiley.","DOI":"10.1002\/9781118439241"},{"key":"ref_102","unstructured":"Rea, L.M., and Parker, R.A. (2014). Designing and Conducting Survey Research: A Comprehensive Guide, Wiley."},{"key":"#cr-split#-ref_103.1","unstructured":"European Union (2016). Regulation"},{"key":"#cr-split#-ref_103.2","unstructured":"(EU) 2016\/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95\/46\/EC (General Data Protection Regulation)."},{"key":"ref_104","unstructured":"Kaplan, D. (2008). Structural Equation Modeling: Foundations and Extensions, SAGE."},{"key":"ref_105","unstructured":"Kline, R.B. (2015). Principles and Practice of Structural Equation Modeling, Guilford Press."},{"key":"ref_106","first-page":"28","article-title":"Validity and Reliability of the Research Instrument; How to Test the Validation of a Questionnaire\/Survey in a Research","volume":"5","author":"Taherdoost","year":"2016","journal-title":"Int. J. Acad. Res. Manag."},{"key":"ref_107","doi-asserted-by":"crossref","unstructured":"Craigen, D., Gerhart, S., and Ralston, T. (1995). Industrial Applications of Formal Methods to Model, Design and Analyze Computer Systems: An International Survey, William Andrew.","DOI":"10.1016\/B978-0-8155-1362-9.50005-6"},{"key":"ref_108","unstructured":"Bowen, J.P., and Nicholls, J.E. (1992, January 14\u201315). An International Survey of Industrial Applications of Formal Methods. Proceedings of the 7th Z User Workshop, London, UK."},{"key":"ref_109","first-page":"328","article-title":"Formal Methods Technology Transfer: Impediments and Innovation","volume":"Volume 962","author":"Lee","year":"1995","journal-title":"Proceedings of the 6th International Conference on Concurrency Theory (CONCUR\u201995)"},{"key":"ref_110","doi-asserted-by":"crossref","first-page":"90","DOI":"10.1109\/32.345825","article-title":"Formal Methods Reality Check: Industrial Usage","volume":"21","author":"Craigen","year":"1995","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_111","doi-asserted-by":"crossref","first-page":"626","DOI":"10.1145\/242223.242257","article-title":"Formal Methods: State of the Art and Future Directions","volume":"28","author":"Clarke","year":"1996","journal-title":"ACM Comput. Surv."},{"key":"ref_112","doi-asserted-by":"crossref","first-page":"19:1","DOI":"10.1145\/1592434.1592436","article-title":"Formal methods: Practice and experience","volume":"41","author":"Woodcock","year":"2009","journal-title":"ACM Comput. Surv."},{"key":"ref_113","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1007\/978-3-319-98938-9_2","article-title":"On the Industrial Uptake of Formal Methods in the Railway Domain: A Survey with Stakeholders","volume":"Volume 11023","author":"Furia","year":"2018","journal-title":"Proceedings of the 14th International Conference on Integrated Formal Methods (iFM\u201918)"},{"key":"ref_114","first-page":"762","article-title":"Adopting Formal Methods in an Industrial Setting: The Railways Case","volume":"Volume 11800","author":"McIver","year":"2019","journal-title":"Proceedings of the 3rd World Congress on Formal Methods: The Next 30 Years (FM\u201919)"},{"key":"ref_115","doi-asserted-by":"crossref","first-page":"4473","DOI":"10.1007\/s10664-020-09836-5","article-title":"Formal Methods in Dependable Systems Engineering: A Survey of Professionals from Europe and North America","volume":"25","author":"Gleirscher","year":"2020","journal-title":"Empir. Softw. Eng."},{"key":"ref_116","unstructured":"Lin\u00e5ker, J., Sulaman, S.M., Maiani de Mello, R., and H\u00f6st, M. (2015). Guidelines for Conducting Surveys in Software Engineering, Department of Computer Science, Lund University. Technical report."},{"key":"ref_117","doi-asserted-by":"crossref","unstructured":"Shull, F., Singer, J., and Sj\u00f8berg, D.I.K. (2008). Personal Opinion Surveys. Guide to Advanced Empirical Software Engineering, Springer.","DOI":"10.1007\/978-1-84800-044-5"},{"key":"ref_118","doi-asserted-by":"crossref","unstructured":"Felderer, M., and Travassos, G.H. (2020). Challenges in Survey Research. Contemporary Empirical Methods in Software Engineering, Springer.","DOI":"10.1007\/978-3-030-32489-6"},{"key":"ref_119","doi-asserted-by":"crossref","first-page":"106240","DOI":"10.1016\/j.infsof.2019.106240","article-title":"An empirically evaluated checklist for surveys in software engineering","volume":"119","author":"Petersen","year":"2020","journal-title":"Inf. Softw. Technol."},{"key":"ref_120","doi-asserted-by":"crossref","first-page":"94:1","DOI":"10.1007\/s10664-021-10072-8","article-title":"Sampling in software engineering research: A critical review and guidelines","volume":"27","author":"Baltes","year":"2022","journal-title":"Empir. Softw. Eng."},{"key":"ref_121","doi-asserted-by":"crossref","unstructured":"Ralph, P., and Tempero, E.D. (2018, January 28\u201329). Construct Validity in Software Engineering Research and Software Metrics. Proceedings of the 22nd ACM International Conference on Evaluation and Assessment in Software Engineering (EASE\u201918), Christchurch, New Zealand.","DOI":"10.1145\/3210459.3210461"},{"key":"ref_122","unstructured":"Heeringa, S.G., West, B.T., and Berglund, P.A. (2020). Applied Survey Data Analysis, Taylor & Francis. Statistics in the Social and Behavioral Sciences."},{"key":"ref_123","doi-asserted-by":"crossref","unstructured":"Agresti, A. (2012). Categorical Data Analysis, Wiley.","DOI":"10.1007\/978-3-642-04898-2_161"},{"key":"ref_124","doi-asserted-by":"crossref","first-page":"2298","DOI":"10.1007\/s10664-016-9451-7","article-title":"Naming the pain in requirements engineering: Contemporary problems, causes, and effects in practice","volume":"22","author":"Wagner","year":"2017","journal-title":"Empir. Softw. Eng."},{"key":"ref_125","doi-asserted-by":"crossref","first-page":"4927","DOI":"10.1007\/s10664-020-09875-y","article-title":"Pandemic programming","volume":"25","author":"Ralph","year":"2020","journal-title":"Empir. Softw. Eng."},{"key":"ref_126","doi-asserted-by":"crossref","first-page":"961","DOI":"10.1016\/j.jss.2007.08.020","article-title":"A survey study of critical success factors in agile software projects","volume":"81","author":"Chow","year":"2008","journal-title":"J. Syst. Softw."},{"key":"ref_127","doi-asserted-by":"crossref","first-page":"2110","DOI":"10.1016\/j.jss.2013.03.084","article-title":"Relevance, benefits, and problems of software modelling and model driven techniques\u2014A survey in the Italian industry","volume":"86","author":"Torchiano","year":"2013","journal-title":"J. Syst. Softw."},{"key":"ref_128","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1007\/s10664-005-1290-x","article-title":"Studying Software Engineers: Data Collection Techniques for Software Field Studies","volume":"10","author":"Lethbridge","year":"2005","journal-title":"Empir. Softw. Eng."},{"key":"ref_129","doi-asserted-by":"crossref","unstructured":"Kaiser, G., and Presmeg, N. (2019). An Introduction to Grounded Theory with a Special Focus on Axial Coding and the Coding Paradigm. Compendium for Early Career Researchers in Mathematics Education, Springer. ICME-13 Monographs.","DOI":"10.1007\/978-3-030-15636-7"},{"key":"ref_130","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1007\/BF00988593","article-title":"Grounded Theory Research: Procedures, Canons, and Evaluative Criteria","volume":"13","author":"Corbin","year":"1990","journal-title":"Qual. Sociol."},{"key":"ref_131","unstructured":"Glaser, B.G. (1978). Theoretical Sensitivity: Advances in the Methodology of Grounded Theory, Sociology Press."},{"key":"ref_132","unstructured":"Charmaz, K. (2006). Constructing Grounded Theory: A Practical Guide through Qualitative Analysis, SAGE."},{"key":"ref_133","doi-asserted-by":"crossref","unstructured":"Hoda, R. (2021). Socio-Technical Grounded Theory for Software Engineering. IEEE Trans. Softw. Eng.","DOI":"10.1109\/ICSE-Companion52605.2021.00139"},{"key":"ref_134","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1191\/1478088706qp063oa","article-title":"Using thematic analysis in psychology","volume":"3","author":"Braun","year":"2006","journal-title":"Qual. Res. Psychol."},{"key":"ref_135","doi-asserted-by":"crossref","unstructured":"Cruzes, D.S., and Dyb\u00e5, T. (2011, January 22\u201323). Recommended Steps for Thematic Synthesis in Software Engineering. Proceedings of the 5th International Symposium on Empirical Software Engineering and Measurement (ESEM\u201911), IEEE, Banff, AB, Canada.","DOI":"10.1109\/ESEM.2011.36"},{"key":"ref_136","unstructured":"Denzin, N.K., and Lincoln, Y.S. (1994). Competing Paradigms in Qualitative Research. Handbook of Qualitative Research, SAGE. Chapter 6."},{"key":"ref_137","doi-asserted-by":"crossref","first-page":"324","DOI":"10.4103\/2249-4863.161306","article-title":"Validity, reliability, and generalizability in qualitative research","volume":"4","author":"Leung","year":"2015","journal-title":"J. Family Med. Prim. Care"},{"key":"ref_138","doi-asserted-by":"crossref","first-page":"305","DOI":"10.1080\/14780887.2020.1780357","article-title":"The pursuit of quality in grounded theory","volume":"18","author":"Charmaz","year":"2021","journal-title":"Qual. Res. Psychol."},{"key":"ref_139","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1016\/S0950-5849(00)00166-X","article-title":"Practitioners\u2019 views on the use of formal methods: An industrial survey by structured interview","volume":"43","author":"Snook","year":"2001","journal-title":"Inf. Softw. Technol."},{"key":"ref_140","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/3-540-40891-6_19","article-title":"Formal Methods Diffusion: Past Lessons and Future Prospects","volume":"Volume 1943","author":"Koornneef","year":"2000","journal-title":"Proceedings of the 19th International Conference on Computer Safety, Reliability and Security (SAFECOMP\u201900)"},{"key":"ref_141","unstructured":"Bryman, A. (2016). Social Research Methods, Oxford University Press."},{"key":"ref_142","doi-asserted-by":"crossref","first-page":"786","DOI":"10.1109\/TSE.2016.2519887","article-title":"The Role of Ethnographic Studies in Empirical Software Engineering","volume":"42","author":"Sharp","year":"2016","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_143","doi-asserted-by":"crossref","unstructured":"Zhang, H., Huang, X., Zhou, X., Huang, H., and Babar, M.A. (2019, January 26\u201330). Ethnographic Research in Software Engineering: A Critical Review and Checklist. Proceedings of the 27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE\u201919), Tallinn, Estonia.","DOI":"10.1145\/3338906.3338976"},{"key":"ref_144","unstructured":"Salda\u00f1a, J. (2021). The Coding Manual for Qualitative Researchers, SAGE."},{"key":"ref_145","doi-asserted-by":"crossref","unstructured":"Stol, K.J., Ralph, P., and Fitzgerald, B. (2016, January 14\u201322). Grounded Theory in Software Engineering Research: A Critical Review and Guidelines. Proceedings of the ACM 38th International Conference on Software Engineering (ICSE\u201916), Austin, TX, USA.","DOI":"10.1145\/2884781.2884833"},{"key":"ref_146","doi-asserted-by":"crossref","first-page":"315","DOI":"10.1007\/s00766-019-00319-8","article-title":"The impact of requirements on systems development speed: A multiple-case study in automotive","volume":"24","author":"Knauss","year":"2019","journal-title":"Requir. Eng."},{"key":"ref_147","doi-asserted-by":"crossref","unstructured":"Yang, N., Cuijpers, P.J.L., Schiffelers, R.R.H., Lukkien, J., and Serebrenik, A. (2021, January 25\u201328). An Interview Study of how Developers use Execution Logs in Embedded Software Engineering. Proceedings of the 43rd IEEE International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP\u201921), Madrid, Spain.","DOI":"10.1109\/ICSE-SEIP52600.2021.00015"},{"key":"ref_148","doi-asserted-by":"crossref","first-page":"46434","DOI":"10.1109\/ACCESS.2019.2909093","article-title":"Information Flow in Software Testing\u2014An Interview Study With Embedded Software Engineering Practitioners","volume":"7","author":"Strandberg","year":"2019","journal-title":"IEEE Access"},{"key":"ref_149","doi-asserted-by":"crossref","first-page":"1579","DOI":"10.1109\/TSE.2020.3025317","article-title":"Real World Scrum A Grounded Theory of Variations in Practice","volume":"48","author":"Masood","year":"2022","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_150","doi-asserted-by":"crossref","first-page":"106672","DOI":"10.1016\/j.infsof.2021.106672","article-title":"The organization of software teams in the quest for continuous delivery: A grounded theory approach","volume":"139","author":"Leite","year":"2021","journal-title":"Inf. Softw. Technol."},{"key":"ref_151","doi-asserted-by":"crossref","first-page":"466","DOI":"10.1177\/002188637100700404","article-title":"A Group Process Model for Problem Identification and Program Planning","volume":"7","author":"Delbecq","year":"1971","journal-title":"J. Appl. Behav. Sci."},{"key":"ref_152","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1111\/j.1365-2648.2006.03716.x","article-title":"Consulting the oracle: Ten lessons from using the Delphi technique in nursing research","volume":"53","author":"Keeney","year":"2006","journal-title":"J. Adv. Nurs."},{"key":"ref_153","unstructured":"Krueger, R.A., and Casey, M.A. (2014). Focus Groups: A Practical Guide for Applied Research, SAGE."},{"key":"ref_154","doi-asserted-by":"crossref","first-page":"463","DOI":"10.1080\/03098260600927575","article-title":"A Practical Guide to Focus-Group Research","volume":"30","author":"Breen","year":"2006","journal-title":"J. Geogr. High. Educ."},{"key":"ref_155","doi-asserted-by":"crossref","unstructured":"Shull, F., Singer, J., and Sj\u00f8berg, D.I.K. (2008). The Focus Group Method as an Empirical Tool in Software Engineering. Guide to Advanced Empirical Software Engineering, Springer.","DOI":"10.1007\/978-1-84800-044-5"},{"key":"ref_156","doi-asserted-by":"crossref","first-page":"458","DOI":"10.1287\/mnsc.9.3.458","article-title":"An Experimental Application of the DELPHI Method to the Use of Experts","volume":"9","author":"Dalkey","year":"1963","journal-title":"Manag. Sci."},{"key":"ref_157","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1108\/14777260310469319","article-title":"Delphi: Myths and reality","volume":"17","author":"Mullen","year":"2003","journal-title":"J. Health Organ. Manag."},{"key":"ref_158","doi-asserted-by":"crossref","first-page":"100867","DOI":"10.1016\/j.ienj.2020.100867","article-title":"Use of the Delphi method to generate guidance in emergency nursing practice: A systematic review","volume":"56","author":"Varndell","year":"2021","journal-title":"Int. Emerg. Nurs."},{"key":"ref_159","first-page":"655","article-title":"How to use the nominal group and Delphi techniques","volume":"38","author":"McMillan","year":"2016","journal-title":"Int. J. Clin. Pharm."},{"key":"ref_160","unstructured":"Dunham, R.B. (1998). Nominal Group Technique: A Users\u2019 Guide, University of Wisconsin-Madison. Technical report."},{"key":"ref_161","unstructured":"Shestopalov, S. (2019). Organizing Brainstorming Workshops: A Designer\u2019s Guide. Smashing Mag."},{"key":"ref_162","unstructured":"Fairley, R. (1985). Software Engineering Concepts, McGraw-Hill."},{"key":"ref_163","doi-asserted-by":"crossref","first-page":"398","DOI":"10.1016\/j.im.2013.05.005","article-title":"Understanding the most critical skills for managing IT projects: A Delphi study of IT project managers","volume":"50","author":"Keil","year":"2013","journal-title":"Inf. Manag."},{"key":"ref_164","doi-asserted-by":"crossref","first-page":"136","DOI":"10.1016\/j.jss.2014.12.010","article-title":"Soft competency requirements in requirements engineering, software design, implementation, and testing","volume":"101","author":"Holtkamp","year":"2015","journal-title":"J. Syst. Softw."},{"key":"ref_165","doi-asserted-by":"crossref","unstructured":"Groeneveld, W., Jacobs, H., Vennekens, J., and Aerts, K. (2020, January 11\u201314). Non-cognitive Abilities of Exceptional Software Engineers: A Delphi Study. Proceedings of the 51st Technical Symposium on Computer Science Education (SIGCSE\u201920), ACM, Portland, OR, USA.","DOI":"10.1145\/3328778.3366811"},{"key":"ref_166","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1016\/j.ienj.2018.11.003","article-title":"Emergency department registered nurses\u2019 disaster medicine competencies. An exploratory study utilizing a modified Delphi technique","volume":"43","author":"Murphy","year":"2019","journal-title":"Int. Emerg. Nurs."},{"key":"ref_167","doi-asserted-by":"crossref","first-page":"188","DOI":"10.1111\/j.1440-172X.2012.02017.x","article-title":"Nominal group technique: An effective method for obtaining group consensus","volume":"18","author":"Harvey","year":"2012","journal-title":"Int. J. Nurs. Pract."},{"key":"ref_168","doi-asserted-by":"crossref","unstructured":"Abbas, M., Ferrari, A., Shatnawi, A., Enoiu, E., Saadatmand, M., and Sundmark, D. (2022). On the relationship between similar requirements and similar software: A case study in the railway domain. Requir. Eng.","DOI":"10.1007\/s00766-021-00370-4"},{"key":"ref_169","doi-asserted-by":"crossref","unstructured":"Martakis, A., and Daneva, M. (2013, January 29\u201331). Handling requirements dependencies in agile projects: A focus group with agile software development practitioners. Proceedings of the 7th International Conference on Research Challenges in Information Science (RCIS\u201913), IEEE, Paris, France.","DOI":"10.1109\/RCIS.2013.6577679"},{"key":"ref_170","doi-asserted-by":"crossref","first-page":"e1990","DOI":"10.1002\/smr.1990","article-title":"Requirements elicitation and refinement in collaborative research projects","volume":"30","author":"Ferrari","year":"2018","journal-title":"J. Softw. Evol. Process."},{"key":"ref_171","doi-asserted-by":"crossref","first-page":"106514","DOI":"10.1016\/j.infsof.2021.106514","article-title":"Case Study Research in Software Engineering\u2014It is a Case, and it is a Study, but is it a Case Study?","volume":"133","author":"Wohlin","year":"2021","journal-title":"Inf. Softw. Technol."},{"key":"ref_172","doi-asserted-by":"crossref","unstructured":"Staron, M. (2020). Action Research as Research Methodology in Software Engineering. Action Research in Software Engineering: Theory and Applications, Springer. Chapter 2.","DOI":"10.1007\/978-3-030-32610-4"},{"key":"ref_173","doi-asserted-by":"crossref","unstructured":"Wieringa, R.J. (2014). Design Science Methodology for Information Systems and Software Engineering, Springer.","DOI":"10.1007\/978-3-662-43839-8"},{"key":"ref_174","doi-asserted-by":"crossref","unstructured":"Runeson, P., H\u00f6st, M., Rainer, A., and Regnell, B. (2012). Case Study Research in Software Engineering: Guidelines and Examples, Wiley.","DOI":"10.1002\/9781118181034"},{"key":"ref_175","doi-asserted-by":"crossref","first-page":"136","DOI":"10.1016\/j.scico.2014.11.013","article-title":"Six strategies for generalizing software engineering theories","volume":"101","author":"Wieringa","year":"2015","journal-title":"Sci. Comput. Program."},{"key":"ref_176","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1016\/S1474-6670(17)36335-8","article-title":"Experiences with the Application of Discrete Formal Methods to the Development of Engine Control Software","volume":"31","author":"Galloway","year":"1998","journal-title":"IFAC Proc. Vol."},{"key":"ref_177","doi-asserted-by":"crossref","first-page":"430","DOI":"10.1007\/978-3-319-96142-2_26","article-title":"Continuous Formal Verification of Amazon s2n","volume":"Volume 10982","author":"Chockler","year":"2018","journal-title":"Proceedings of the 30th International Conference on Computer Aided Verification (CAV\u201918)"},{"key":"ref_178","doi-asserted-by":"crossref","first-page":"683","DOI":"10.1007\/s00165-010-0172-1","article-title":"Automated property verification for large scale B models with ProB","volume":"23","author":"Leuschel","year":"2011","journal-title":"Form. Asp. Comput."},{"key":"ref_179","doi-asserted-by":"crossref","first-page":"828","DOI":"10.1016\/j.scico.2012.04.003","article-title":"The Metr\u00f4 Rio case study","volume":"78","author":"Ferrari","year":"2013","journal-title":"Sci. Comput. Program."},{"key":"ref_180","doi-asserted-by":"crossref","first-page":"126","DOI":"10.1016\/j.is.2014.11.007","article-title":"Efficient formalization of railway interlocking data in RailML","volume":"49","author":"Bosschaart","year":"2015","journal-title":"Inf. Syst."},{"key":"ref_181","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1016\/j.jss.2016.09.027","article-title":"Supporting pattern-based dependability engineering via model-driven development: Approach, tool-support and empirical validation","volume":"122","author":"Hamid","year":"2016","journal-title":"J. Syst. Softw."},{"key":"ref_182","first-page":"202","article-title":"Property-Based Modelling and Validation of a CBTC Zone Controller in Event-B","volume":"Volume 11495","author":"Lecomte","year":"2019","journal-title":"Proceedings of the 3rd International Conference on Reliability, Safety, and Security of Railway Systems: Modelling, Analysis, Verification, and Certification (RSSRail\u201919)"},{"key":"ref_183","first-page":"369","article-title":"M\u00e9t\u00e9or: A Successful Application of B in a Large Project","volume":"Volume 1708","author":"Wing","year":"1999","journal-title":"Proceedings of the 1st World Congress on Formal Methods in the Development of Computing Systems (FM\u201999)"},{"key":"ref_184","doi-asserted-by":"crossref","unstructured":"Johannesson, P., and Perjons, E. (2014). An Introduction to Design Science, Springer.","DOI":"10.1007\/978-3-319-10632-8"},{"key":"ref_185","doi-asserted-by":"crossref","first-page":"110699","DOI":"10.1016\/j.jss.2020.110699","article-title":"Evaluating and strategizing the onboarding of software developers in large-scale globally distributed projects","volume":"169","author":"Britto","year":"2020","journal-title":"J. Syst. Softw."},{"key":"ref_186","doi-asserted-by":"crossref","first-page":"217","DOI":"10.1016\/j.infsof.2019.06.010","article-title":"DevOps in practice: A multiple case study of five companies","volume":"114","author":"Lwakatare","year":"2019","journal-title":"Inf. Softw. Technol."},{"key":"ref_187","doi-asserted-by":"crossref","first-page":"863","DOI":"10.1109\/TSE.2018.2871058","article-title":"The Adoption of JavaScript Linters in Practice: A Case Study on ESLint","volume":"46","author":"Aniche","year":"2020","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_188","doi-asserted-by":"crossref","first-page":"3684","DOI":"10.1007\/s10664-018-9596-7","article-title":"Detecting requirements defects with NLP patterns: An industrial experience in the railway domain","volume":"23","author":"Ferrari","year":"2018","journal-title":"Empir. Softw. Eng."},{"key":"ref_189","doi-asserted-by":"crossref","first-page":"220","DOI":"10.1007\/s10664-019-09769-8","article-title":"Recognizing lines of code violating company-specific coding guidelines using machine learning","volume":"25","author":"Ochodek","year":"2020","journal-title":"Empir. Softw. Eng."},{"key":"ref_190","doi-asserted-by":"crossref","first-page":"106433","DOI":"10.1016\/j.infsof.2020.106433","article-title":"A Method to Estimate Software Strategic Indicators in Software Development: An Industrial Application","volume":"129","author":"Manzano","year":"2021","journal-title":"Inf. Softw. Technol."},{"key":"ref_191","doi-asserted-by":"crossref","first-page":"571","DOI":"10.1016\/j.jss.2006.07.009","article-title":"Lessons from applying the systematic literature review process within the software engineering domain","volume":"80","author":"Brereton","year":"2007","journal-title":"J. Syst. Softw."},{"key":"ref_192","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1016\/j.infsof.2008.09.009","article-title":"Systematic literature reviews in software engineering - A systematic literature review","volume":"51","author":"Kitchenham","year":"2009","journal-title":"Inf. Softw. Technol."},{"key":"ref_193","doi-asserted-by":"crossref","unstructured":"Garousi, V., Felderer, M., and M\u00e4ntyl\u00e4, M.V. (2016, January 1\u20133). The need for multivocal literature reviews in software engineering: Complementing systematic literature reviews with grey literature. Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering (EASE\u201916), ACM, Limerick, Ireland.","DOI":"10.1145\/2915970.2916008"},{"key":"ref_194","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1016\/j.infsof.2018.09.006","article-title":"Guidelines for including grey literature and conducting multivocal literature reviews in software engineering","volume":"106","author":"Garousi","year":"2019","journal-title":"Inf. Softw. Technol."},{"key":"ref_195","doi-asserted-by":"crossref","unstructured":"Ferrari, A., and ter Beek, M.H. (2022). Formal Methods in Railways: A Systematic Mapping Study. ACM Comput. Surv.","DOI":"10.1145\/3520480"},{"key":"ref_196","doi-asserted-by":"crossref","first-page":"370","DOI":"10.1007\/978-3-030-99619-2_35","article-title":"Security, Power Consumption and Simulations in IoT Device Networks: A Systematic Review","volume":"Volume 451","author":"Barolli","year":"2022","journal-title":"Proceedings of the 36th International Conference on Advanced Information Networking and Applications (AINA\u201922)"},{"key":"ref_197","first-page":"281","article-title":"The AVISPA Tool for the Automated Validation of Internet Security Protocols and Applications","volume":"Volume 3576","author":"Etessami","year":"2005","journal-title":"Proceedings of the 17th International Conference on Computer Aided Verification (CAV\u201905)"},{"key":"ref_198","doi-asserted-by":"crossref","first-page":"6702","DOI":"10.1002\/cpe.6702","article-title":"A review on security requirements specification by formal methods","volume":"34","author":"Mishra","year":"2022","journal-title":"Concurr. Comput. Pract. Exp."},{"key":"ref_199","doi-asserted-by":"crossref","first-page":"1603","DOI":"10.1007\/s10845-021-01753-8","article-title":"A systematic mapping of semi-formal and formal methods in requirements engineering of industrial Cyber-Physical systems","volume":"33","author":"Zahid","year":"2022","journal-title":"J. Intell. Manuf."},{"key":"ref_200","unstructured":"Durelli, R.S., and Durelli, V.H.S. (2012, January 12\u201314). A systematic review on mining techniques for crosscutting concerns. Proceedings of the IX Experimental Software Engineering Latin American Workshop (ESELAW\u201912), Bras\u00edlia, Brazil."},{"key":"ref_201","doi-asserted-by":"crossref","unstructured":"Weyns, D., Iftikhar, M.U., de la Iglesia, D.G., and Ahmad, T. (2012, January 27\u201329). A survey of formal methods in self-adaptive systems. Proceedings of the 5th ACM International C* Conference on Computer Science & Software Engineering (C3S2E\u201912), Montreal, QC, Canada.","DOI":"10.1145\/2347583.2347592"},{"key":"ref_202","doi-asserted-by":"crossref","first-page":"e1943","DOI":"10.1002\/smr.1943","article-title":"A systematic literature review of the use of formal methods in medical software systems","volume":"30","author":"Bonfanti","year":"2018","journal-title":"J. Softw. Evol. Process."},{"key":"ref_203","doi-asserted-by":"crossref","first-page":"2350","DOI":"10.1002\/spe.2634","article-title":"Evaluating the suitability of state-based formal methods for industrial deployment","volume":"48","author":"Mashkoor","year":"2018","journal-title":"Softw. Pract. Exp."},{"key":"ref_204","doi-asserted-by":"crossref","first-page":"4797","DOI":"10.1109\/ACCESS.2020.3048047","article-title":"Software Verification and Validation of Safe Autonomous Cars: A Systematic Literature Review","volume":"9","author":"Rajabli","year":"2021","journal-title":"IEEE Access"},{"key":"ref_205","doi-asserted-by":"crossref","first-page":"218","DOI":"10.1007\/978-3-030-71374-4_12","article-title":"Teaching Formal Methods in Academia: A Systematic Literature Review","volume":"Volume 1301","author":"Cerone","year":"2021","journal-title":"Proceedings of the 1st International Workshop on Formal Methods \u2013 Fun for Everybody (FMFun\u201919)"},{"key":"ref_206","unstructured":"Kitchenham, B. (2004). Procedures for Performing Systematic Reviews, Keele University. Technical Report TR\/SE-0401."},{"key":"ref_207","doi-asserted-by":"crossref","unstructured":"Visaggio, G., Baldassarre, M.T., Linkman, S.G., and Turner, M. (2008, January 26\u201327). Systematic Mapping Studies in Software Engineering. Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering (EASE\u201908), Bari, Italy. Workshops in Computing.","DOI":"10.14236\/ewic\/EASE2008.0"},{"key":"ref_208","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.infsof.2015.03.007","article-title":"Guidelines for conducting systematic mapping studies in software engineering: An update","volume":"64","author":"Petersen","year":"2015","journal-title":"Inf. Softw. Technol."},{"key":"ref_209","doi-asserted-by":"crossref","first-page":"625","DOI":"10.1016\/j.infsof.2010.12.010","article-title":"Identifying relevant studies in software engineering","volume":"53","author":"Zhang","year":"2011","journal-title":"Inf. Softw. Technol."},{"key":"ref_210","unstructured":"Shepperd, M.J., Hall, T., and Myrtveit, I. (2014, January 13\u201314). Guidelines for Snowballing in Systematic Literature Studies and a Replication in Software Engineering. Proceedings of the 18th ACM International Conference on Evaluation and Assessment in Software Engineering (EASE\u201914), London, UK."},{"key":"ref_211","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1016\/j.infsof.2018.10.006","article-title":"Identifying, categorizing and mitigating threats to validity in software engineering secondary studies","volume":"106","author":"Ampatzoglou","year":"2019","journal-title":"Inf. Softw. Technol."},{"key":"ref_212","doi-asserted-by":"crossref","first-page":"106366","DOI":"10.1016\/j.infsof.2020.106366","article-title":"Guidelines for the search strategy to update systematic literature reviews in software engineering","volume":"127","author":"Wohlin","year":"2020","journal-title":"Inf. Softw. Technol."},{"key":"ref_213","first-page":"43:1","article-title":"Analysing app reviews for software engineering: A systematic literature review","volume":"27","author":"Letier","year":"2022","journal-title":"Empir. Softw. Eng."},{"key":"ref_214","doi-asserted-by":"crossref","first-page":"148","DOI":"10.1016\/j.infsof.2014.06.011","article-title":"A systematic review on the relationship between user involvement and system success","volume":"58","author":"Bano","year":"2015","journal-title":"Inf. Softw. Technol."},{"key":"ref_215","first-page":"37e:1","article-title":"Software Engineering for AI-Based Systems: A Survey","volume":"31","author":"Bogner","year":"2022","journal-title":"ACM Trans. Softw. Eng. Methodol."},{"key":"ref_216","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1007\/s00766-017-0280-z","article-title":"Goal-oriented requirements engineering: An extended systematic mapping study","volume":"24","author":"Horkoff","year":"2019","journal-title":"Requir. Eng."},{"key":"ref_217","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1016\/j.infsof.2017.01.001","article-title":"Software test maturity assessment and test process improvement: A multivocal literature review","volume":"85","author":"Garousi","year":"2017","journal-title":"Inf. Softw. Technol."},{"key":"ref_218","doi-asserted-by":"crossref","first-page":"110708","DOI":"10.1016\/j.jss.2020.110708","article-title":"Function-as-a-Service performance evaluation: A multivocal literature review","volume":"170","author":"Scheuner","year":"2020","journal-title":"J. Syst. Softw."}],"container-title":["Software"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2674-113X\/1\/4\/17\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:38:42Z","timestamp":1760143122000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2674-113X\/1\/4\/17"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,24]]},"references-count":219,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["software1040017"],"URL":"https:\/\/doi.org\/10.3390\/software1040017","relation":{},"ISSN":["2674-113X"],"issn-type":[{"type":"electronic","value":"2674-113X"}],"subject":[],"published":{"date-parts":[[2022,9,24]]}}}