{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,20]],"date-time":"2026-02-20T18:23:57Z","timestamp":1771611837418,"version":"3.50.1"},"reference-count":87,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2023,3,6]],"date-time":"2023-03-06T00:00:00Z","timestamp":1678060800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,3,6]],"date-time":"2023-03-06T00:00:00Z","timestamp":1678060800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000781","name":"European Research Council","doi-asserted-by":"publisher","award":["864972"],"award-info":[{"award-number":["864972"]}],"id":[{"id":"10.13039\/501100000781","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000781","name":"European Research Council","doi-asserted-by":"publisher","award":["864972"],"award-info":[{"award-number":["864972"]}],"id":[{"id":"10.13039\/501100000781","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000781","name":"European Research Council","doi-asserted-by":"publisher","award":["864972"],"award-info":[{"award-number":["864972"]}],"id":[{"id":"10.13039\/501100000781","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000781","name":"European Research Council","doi-asserted-by":"publisher","award":["864972"],"award-info":[{"award-number":["864972"]}],"id":[{"id":"10.13039\/501100000781","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000781","name":"European Research Council","doi-asserted-by":"publisher","award":["864972"],"award-info":[{"award-number":["864972"]}],"id":[{"id":"10.13039\/501100000781","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100010253","name":"Secretar\u00eda de Ciencia y T\u00e9cnica, Universidad de Buenos Aires","doi-asserted-by":"publisher","award":["UBACyT 2020 20020190100233BA"],"award-info":[{"award-number":["UBACyT 2020 20020190100233BA"]}],"id":[{"id":"10.13039\/501100010253","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100021778","name":"Agencia Nacional de Promoci\u00f3n de la Investigaci\u00f3n, el Desarrollo Tecnol\u00f3gico y la Innovaci\u00f3n","doi-asserted-by":"publisher","award":["PICT-2019-01793"],"award-info":[{"award-number":["PICT-2019-01793"]}],"id":[{"id":"10.13039\/501100021778","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Kristiania University College"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Software Qual J"],"published-print":{"date-parts":[[2023,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Research in software testing often involves the development of software prototypes. Like any piece of software, there are challenges in the development, use and verification of such tools. However, some challenges are rather specific to this problem domain. For example, often these tools are developed by PhD students straight out of bachelor\/master degrees, possibly lacking any industrial experience in software development. Prototype tools are used to carry out empirical studies, possibly studying different parameters of novel designed algorithms. Software scaffolding is needed to run large sets of experiments efficiently. Furthermore, when using AI-based techniques like evolutionary algorithms, care needs to be taken to deal with their randomness, which further complicates their verification. The aforementioned represent some of the challenges we have identified for this domain. In this paper, we report on our experience in building the open-source <jats:sc>EvoMaster<\/jats:sc> tool, which aims at system-level test case generation for enterprise applications. Many of the challenges we faced would be common to any researcher needing to build software testing tool prototypes. Therefore, one goal is that our shared experience here will boost the research community, by providing concrete solutions to many development challenges in the building of such kind of research prototypes. Ultimately, this will lead to increase the impact of scientific research on industrial practice.<\/jats:p>","DOI":"10.1007\/s11219-023-09620-w","type":"journal-article","created":{"date-parts":[[2023,3,6]],"date-time":"2023-03-06T03:02:53Z","timestamp":1678071773000},"page":"947-990","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Building an open-source system test generation tool: lessons learned and empirical analyses with EvoMaster"],"prefix":"10.1007","volume":"31","author":[{"given":"Andrea","family":"Arcuri","sequence":"first","affiliation":[]},{"given":"Man","family":"Zhang","sequence":"additional","affiliation":[]},{"given":"Asma","family":"Belhadi","sequence":"additional","affiliation":[]},{"given":"Bogdan","family":"Marculescu","sequence":"additional","affiliation":[]},{"given":"Amid","family":"Golmohammadi","sequence":"additional","affiliation":[]},{"given":"Juan Pablo","family":"Galeotti","sequence":"additional","affiliation":[]},{"given":"Susruthan","family":"Seran","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,3,6]]},"reference":[{"key":"9620_CR1","doi-asserted-by":"crossref","unstructured":"Ahmed, K., Lis, M., & Rubin, J. (2021). Slicer4j: A dynamic slicer for java. ESEC\/FSE 2021 (p. 1570-1574). New York, NY, USA: Association for Computing Machinery.","DOI":"10.1145\/3468264.3473123"},{"key":"9620_CR2","doi-asserted-by":"crossref","unstructured":"Arcaini, P., Bombarda, A., Bonfanti, S., & Gargantini, A. (2021). Roby: a tool for robustness analysis of neural network classifiers. In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST) (pp. 442\u2013447). IEEE.","DOI":"10.1109\/ICST49551.2021.00057"},{"key":"9620_CR3","doi-asserted-by":"crossref","unstructured":"Arcuri, A. (2017a). Many Independent Objective (MIO) Algorithm for Test Suite Generation. In International Symposium on Search Based Software Engineering (SSBSE) (pp. 3\u201317).","DOI":"10.1007\/978-3-319-66299-2_1"},{"key":"9620_CR4","doi-asserted-by":"crossref","unstructured":"Arcuri, A. (2017b). RESTful API Automated Test Case Generation. In IEEE International Conference on Software Quality, Reliability and Security (QRS) (pp. 9\u201320). IEEE.","DOI":"10.1109\/QRS.2017.11"},{"key":"9620_CR5","doi-asserted-by":"crossref","unstructured":"Arcuri, A. (2018a). EvoMaster: Evolutionary Multi-context Automated System Test Generation. In IEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE.","DOI":"10.1109\/ICST.2018.00046"},{"key":"9620_CR6","doi-asserted-by":"crossref","unstructured":"Arcuri, A. (2018b). An experience report on applying software testing academic results in industry: we need usable automated test generation. Empirical Software Engineering, 23, 1959\u20131981.","DOI":"10.1007\/s10664-017-9570-9"},{"key":"9620_CR7","doi-asserted-by":"crossref","unstructured":"Arcuri, A. (2018c). Test suite generation with the Many Independent Objective (MIO) algorithm. Information and Software Technology, 104, 195\u2013206.","DOI":"10.1016\/j.infsof.2018.05.003"},{"key":"9620_CR8","doi-asserted-by":"crossref","unstructured":"Arcuri, A. (2019). Restful api automated test case generation with evomaster. ACM Transactions on Software Engineering and Methodology (TOSEM), 28, 3.","DOI":"10.1145\/3293455"},{"key":"9620_CR9","doi-asserted-by":"publisher","first-page":"72","DOI":"10.1109\/MS.2020.3013820","volume":"38","author":"A Arcuri","year":"2020","unstructured":"Arcuri, A. (2020). Automated black-and white-box testing of restful apis with evomaster. IEEE Software, 38, 72\u201378.","journal-title":"IEEE Software"},{"key":"9620_CR10","doi-asserted-by":"publisher","first-page":"219","DOI":"10.1002\/stvr.1486","volume":"24","author":"A Arcuri","year":"2014","unstructured":"Arcuri, A., & Briand, L. (2014). A Hitchhiker\u2019s Guide to Statistical Tests for Assessing Randomized Algorithms in Software Engineering. Software Testing. Verification and Reliability (STVR), 24, 219\u2013250.","journal-title":"Verification and Reliability (STVR)"},{"key":"9620_CR11","doi-asserted-by":"publisher","first-page":"594","DOI":"10.1007\/s10664-013-9249-9","volume":"18","author":"A Arcuri","year":"2013","unstructured":"Arcuri, A., & Fraser, G. (2013). Parameter tuning or default values? an empirical investigation in search-based software engineering. Empirical Software Engineering (EMSE), 18, 594\u2013623.","journal-title":"Empirical Software Engineering (EMSE)"},{"key":"9620_CR12","doi-asserted-by":"crossref","unstructured":"Arcuri, A., & Galeotti, J.\u00a0P. (2019). Sql data generation to enhance search-based system testing. In Proceedings of the Genetic and Evolutionary Computation Conference GECCO \u201919 (p. 1390-1398). New York, NY, USA: Association for Computing Machinery.","DOI":"10.1145\/3321707.3321732"},{"key":"9620_CR13","doi-asserted-by":"crossref","unstructured":"Arcuri, A., & Galeotti, J. P. (2020a). Handling sql databases in automated system test generation. ACM Transactions on Software Engineering and Methodology (TOSEM), 29, 1\u201331.","DOI":"10.1145\/3391533"},{"key":"9620_CR14","doi-asserted-by":"crossref","unstructured":"Arcuri, A., & Galeotti, J.\u00a0P. (2020b). Testability transformations for existing apis. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST) (pp. 153\u2013163). IEEE.","DOI":"10.1109\/ICST46399.2020.00025"},{"key":"9620_CR15","doi-asserted-by":"crossref","unstructured":"Arcuri, A., & Galeotti, J.\u00a0P. (2021a). Enhancing Search-based Testing with Testability Transformations for Existing APIs. ACM Transactions on Software Engineering and Methodology (TOSEM), 31, 1\u201334.","DOI":"10.1145\/3477271"},{"key":"9620_CR16","doi-asserted-by":"crossref","unstructured":"Arcuri, A., & Galeotti, J.\u00a0P. (2021b). Enhancing search-based testing with testability transformations for existing apis. ACM Transactions on Software Engineering and Methodology (TOSEM), 31, 1\u201334.","DOI":"10.1145\/3477271"},{"key":"9620_CR17","doi-asserted-by":"crossref","unstructured":"Arcuri, A., Galeotti, J.\u00a0P., Marculescu, B., & Zhang, M. (2021). Evomaster: A search-based system test generation tool. Journal of Open Source Software, 6, 2153.","DOI":"10.21105\/joss.02153"},{"key":"9620_CR18","doi-asserted-by":"publisher","first-page":"258","DOI":"10.1109\/TSE.2011.121","volume":"38","author":"A Arcuri","year":"2012","unstructured":"Arcuri, A., Iqbal, M. Z., & Briand, L. (2012). Random testing: Theoretical results and practical implications. IEEE Transactions on Software Engineering (TSE), 38, 258\u2013277.","journal-title":"IEEE Transactions on Software Engineering (TSE)"},{"key":"9620_CR19","doi-asserted-by":"publisher","unstructured":"Arcuri, A., ZhangMan, asmab89, Bogdan, Gol, A., Galeotti, J.\u00a0P., Seran, L\u00f3pez, A.\u00a0M., Aldasoro, A., Panichella, A., & Niemeyer, K. (2022). Emresearch\/evomaster: https:\/\/doi.org\/10.5281\/zenodo.6651631","DOI":"10.5281\/zenodo.6651631"},{"key":"9620_CR20","doi-asserted-by":"crossref","unstructured":"Atlidakis, V., Godefroid, P., & Polishchuk, M. (2019). Restler: Stateful REST API fuzzing. In ACM\/IEEE International Conference on Software Engineering (ICSE) (p. 748-758).","DOI":"10.1109\/ICSE.2019.00083"},{"key":"9620_CR21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3182657","volume":"51","author":"R Baldoni","year":"2018","unstructured":"Baldoni, R., Coppa, E., D\u2019elia, D. C., Demetrescu, C., & Finocchi, I. (2018). A survey of symbolic execution techniques. ACM Computing Surveys (CSUR), 51, 1\u201339.","journal-title":"ACM Computing Surveys (CSUR)"},{"key":"9620_CR22","doi-asserted-by":"crossref","unstructured":"Belhadi, A., Zhang, M., & Arcuri, A. (2022). Evolutionary-based Automated Testing for GraphQL APIs. In Genetic and Evolutionary Computation Conference (GECCO).","DOI":"10.1145\/3520304.3528952"},{"key":"9620_CR23","doi-asserted-by":"crossref","unstructured":"Bertolino, A. (2007). Software testing research: Achievements, challenges, dreams. In Future of Software Engineering, 2007. FOSE\u201907 (pp. 85\u2013103). IEEE.","DOI":"10.1109\/FOSE.2007.25"},{"key":"9620_CR24","doi-asserted-by":"crossref","unstructured":"Bertolino, A., De\u00a0Angelis, G., Miranda, B., & Tonella, P. (2020). Run java applications and test them in-vivo meantime. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST) (pp. 454\u2013459). IEEE.","DOI":"10.1109\/ICST46399.2020.00061"},{"key":"9620_CR25","doi-asserted-by":"publisher","first-page":"66","DOI":"10.1145\/1646353.1646374","volume":"53","author":"A Bessey","year":"2010","unstructured":"Bessey, A., Block, K., Chelf, B., Chou, A., Fulton, B., Hallem, S., Henri-Gros, C., Kamsky, A., McPeak, S., & Engler, D. (2010). A few billion lines of code later: using static analysis to find bugs in the real world. Communications of the ACM, 53, 66\u201375.","journal-title":"Communications of the ACM"},{"key":"9620_CR26","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10009-017-0469-y","volume":"21","author":"D Beyer","year":"2019","unstructured":"Beyer, D., L\u00f6we, S., & Wendler, P. (2019). Reliable benchmarking: requirements and solutions. International Journal on Software Tools for Technology Transfer, 21, 1\u201329.","journal-title":"International Journal on Software Tools for Technology Transfer"},{"key":"9620_CR27","doi-asserted-by":"crossref","unstructured":"Borges, N.\u00a0P., & Zeller, A. (2019). Why does this app need this data? automatic tightening of resource access. In 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST) (pp. 449\u2013456). IEEE.","DOI":"10.1109\/ICST.2019.00054"},{"key":"9620_CR28","unstructured":"Brida, S.\u00a0G., Regis, G., Zheng, G., Bagheri, H., Nguyen, T., Aguirre, N., & Frias, M. (2021). Beafix: An automated repair tool for faulty alloy models. In 2021 36th IEEE\/ACM International Conference on Automated Software Engineering (ASE) (pp. 1213\u20131217). IEEE."},{"key":"9620_CR29","doi-asserted-by":"crossref","unstructured":"Brunetto, M., Denaro, G., Mariani, L., & Pezz\u00e8, M. (2021). On introducing automatic test case generation in practice: A success story and lessons learned. Journal of Systems and Software, 176, 110933.","DOI":"10.1016\/j.jss.2021.110933"},{"key":"9620_CR30","doi-asserted-by":"crossref","unstructured":"Bures, M., Ahmed, B.\u00a0S., Rechtberger, V., Klima, M., Trnka, M., Jaros, M., Bellekens, X., Almog, D., & Herout, P. (2021). Patriot: Iot automated interoperability and integration testing framework. In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST) (pp. 454\u2013459). IEEE.","DOI":"10.1109\/ICST49551.2021.00059"},{"key":"9620_CR31","doi-asserted-by":"crossref","unstructured":"Bures, M., Herout, P., & Ahmed, B.\u00a0S. (2020). Open-source defect injection benchmark testbed for the evaluation of testing. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST) (pp. 442\u2013447). IEEE.","DOI":"10.1109\/ICST46399.2020.00059"},{"key":"9620_CR32","unstructured":"Cadar, C., Dunbar, D., Engler, D.\u00a0R. et\u00a0al. (2008). Klee: unassisted and automatic generation of high-coverage tests for complex systems programs. In OSDI (pp. 209\u2013224). volume\u00a08."},{"key":"9620_CR33","doi-asserted-by":"crossref","unstructured":"Cox, H. (2021). Differential coverage:: automating coverage analysis. In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST) (pp. 424\u2013429). IEEE.","DOI":"10.1109\/ICST49551.2021.00054"},{"key":"9620_CR34","unstructured":"EvoMaster. https:\/\/github.com\/EMResearch\/EvoMaster"},{"key":"9620_CR35","unstructured":"Evomaster benchmark (emb). https:\/\/github.com\/EMResearch\/EMB"},{"key":"9620_CR36","doi-asserted-by":"crossref","unstructured":"Fraser, G., & Arcuri, A. (2011). EvoSuite: automatic test suite generation for object-oriented software. In ACM Symposium on the Foundations of Software Engineering (FSE) (pp. 416\u2013419).","DOI":"10.1145\/2025113.2025179"},{"key":"9620_CR37","doi-asserted-by":"publisher","first-page":"2540","DOI":"10.1007\/s10664-019-09711-y","volume":"24","author":"V Garousi","year":"2019","unstructured":"Garousi, V., Pfahl, D., Fernandes, J. M., Felderer, M., M\u00e4ntyl\u00e4, M. V., Shepherd, D., Arcuri, A., Co\u015fkun\u00e7ay, A., & Tekinerdogan, B. (2019). Characterizing industry-academia collaborations in software engineering: evidence from 101 projects. Empirical Software Engineering, 24, 2540\u20132602.","journal-title":"Empirical Software Engineering"},{"key":"9620_CR38","doi-asserted-by":"crossref","unstructured":"Gotz, K., Feldmeier, P., & Fraser, G. (2022). Model-based testing of scratch programs. In 2022 15th IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE.","DOI":"10.1109\/ICST53961.2022.00047"},{"key":"9620_CR39","doi-asserted-by":"crossref","unstructured":"Harman, M., Mansouri, S.\u00a0A., & Zhang, Y. (2012). Search-based software engineering: Trends, techniques and applications. ACM Computing Surveys (CSUR), 45, 11.","DOI":"10.1145\/2379776.2379787"},{"key":"9620_CR40","doi-asserted-by":"crossref","unstructured":"Haryono, S.\u00a0A., Thung, F., Lo, D., Jiang, L., Lawall, J., Kang, H.\u00a0J., Serrano, L., & Muller, G. (2021). Androevolve: Automated update for android deprecated-api usages. In 2021 IEEE\/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (pp. 1\u20134). IEEE.","DOI":"10.1109\/ICSE-Companion52605.2021.00021"},{"key":"9620_CR41","doi-asserted-by":"crossref","unstructured":"He, S., Emmi, M., & Ciocarlie, G. (2020). ct-fuzz: Fuzzing for timing leaks. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST) (pp. 466\u2013471). IEEE.","DOI":"10.1109\/ICST46399.2020.00063"},{"key":"9620_CR42","doi-asserted-by":"crossref","unstructured":"Herlim, R.\u00a0S., Kim, Y., & Kim, M. (2022). Citrus: Automated unit testing tool for real-world c++ programs. In 2022 15th IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE.","DOI":"10.1109\/ICST53961.2022.00046"},{"key":"9620_CR43","doi-asserted-by":"crossref","unstructured":"Heum\u00fcller, R., Nielebock, S., & Ortmeier, F. (2021). Exploit those code reviews! bigger data for deeper learning. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering ESEC\/FSE 2021 (p. 1505-1509). New York, NY, USA: Association for Computing Machinery.","DOI":"10.1145\/3468264.3473110"},{"key":"9620_CR44","doi-asserted-by":"crossref","unstructured":"Hodov\u00e1n, R., & Kiss, \u00c1. (2018). Fuzzinator: an open-source modular random testing framework. In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST) (pp. 416\u2013421). IEEE.","DOI":"10.1109\/ICST.2018.00050"},{"key":"9620_CR45","doi-asserted-by":"crossref","unstructured":"Horlings, E., & Jongmans, S.-S. (2021). Analysis of specifications of multiparty sessions with dcj-lint. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (p. 1590-1594). New York, NY, USA: Association for Computing Machinery.","DOI":"10.1145\/3468264.3473127"},{"key":"9620_CR46","doi-asserted-by":"crossref","unstructured":"Hou, Y., Liu, J., Wang, D., He, J., Fang, C., & Chen, Z. (2021). Taumed: test augmentation of deep learning in medical diagnosis. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis (pp. 674\u2013677).","DOI":"10.1145\/3460319.3469080"},{"key":"9620_CR47","doi-asserted-by":"crossref","unstructured":"Jendele, L., Schwenk, M., Cremarenco, D., Janicijevic, I., & Rybalkin, M. (2019). Efficient automated decomposition of build targets at large-scale. In 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST) (pp. 457\u2013464). IEEE.","DOI":"10.1109\/ICST.2019.00055"},{"key":"9620_CR48","doi-asserted-by":"crossref","unstructured":"Khaireddine, B., Martinez, M., & Mili, A. (2019). Program repair at arbitrary fault depth. In 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST) (pp. 465\u2013472). IEEE.","DOI":"10.1109\/ICST.2019.00056"},{"key":"9620_CR49","doi-asserted-by":"publisher","first-page":"870","DOI":"10.1109\/32.57624","volume":"16","author":"B Korel","year":"1990","unstructured":"Korel, B. (1990). Automated software test data generation. IEEE Transactions on Software Engineering, 16, 870\u2013879.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"9620_CR50","doi-asserted-by":"crossref","unstructured":"Leotta, M., Molinari, A., & Ricca, F. (2022). Assessor: a po-based webdriver test suites generator from selenium ide recordings. In 2022 15th IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE.","DOI":"10.1109\/ICST53961.2022.00045"},{"key":"9620_CR51","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3491038","volume":"31","author":"B Marculescu","year":"2022","unstructured":"Marculescu, B., Zhang, M., & Arcuri, A. (2022). On the faults found in rest apis by automated test generation. ACM Transactions on Software Engineering and Methodology (TOSEM), 31, 1\u201343.","journal-title":"ACM Transactions on Software Engineering and Methodology (TOSEM)"},{"key":"9620_CR52","doi-asserted-by":"crossref","unstructured":"Musco, V., Yin, X., & Neamtiu, I. (2019). Smokeout: An approach for testing clustering implementations. In 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST) (pp. 473\u2013480). IEEE.","DOI":"10.1109\/ICST.2019.00057"},{"key":"9620_CR53","doi-asserted-by":"crossref","unstructured":"Natella, R., & Pham, V.-T. (2021). Profuzzbench: a benchmark for stateful protocol fuzzing. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis (pp. 662\u2013665).","DOI":"10.1145\/3460319.3469077"},{"key":"9620_CR54","doi-asserted-by":"crossref","unstructured":"Nie, C., & Leung, H. (2011). A survey of combinatorial testing. ACM Computing Surveys, 43, 1\u201329. Article 11.","DOI":"10.1145\/1883612.1883618"},{"key":"9620_CR55","doi-asserted-by":"crossref","unstructured":"Olianas, D., Leotta, M., Ricca, F., Biagiola, M., & Tonella, P. (2021). Stile: a tool for parallel execution of e2e web test scripts. In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST) (pp. 460\u2013465). IEEE.","DOI":"10.1109\/ICST49551.2021.00060"},{"key":"9620_CR56","doi-asserted-by":"crossref","unstructured":"Pacheco, C., & Ernst, M.\u00a0D. (2007). Randoop: feedback-directed random testing for Java. In OOPSLA\u201907: Companion to the 22nd ACM SIGPLAN Conference on Object-oriented Programming Systems and Application (pp. 815\u2013816).","DOI":"10.1145\/1297846.1297902"},{"key":"9620_CR57","doi-asserted-by":"crossref","unstructured":"Paduraru, C., Paduraru, M., & Stefanescu, A. (2021). Riverfuzzrl-an open-source tool to experiment with reinforcement learning for fuzzing. In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST) (pp. 430\u2013435). IEEE.","DOI":"10.1109\/ICST49551.2021.00055"},{"key":"9620_CR58","doi-asserted-by":"crossref","unstructured":"Paduraru, C., Paduraru, M., & Stefanescu, A. (2022). Rivergame - a game testing tool using artificial intelligence. In 2022 15th IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE.","DOI":"10.1109\/ICST53961.2022.00048"},{"key":"9620_CR59","doi-asserted-by":"crossref","unstructured":"Paiva, A.\u00a0C., Gon\u00e7alves, M.\u00a0A., & Barros, A.\u00a0R. (2019). Testing android incoming calls. In 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST) (pp. 441\u2013448). IEEE.","DOI":"10.1109\/ICST.2019.00053"},{"key":"9620_CR60","doi-asserted-by":"crossref","unstructured":"Panichella, S., Gambi, A., Zampetti, F., & Riccio, V. (2021). Sbst tool competition 2021. In 2021 IEEE\/ACM 14th International Workshop on Search-Based Software Testing (SBST) (pp. 20\u201327). IEEE.","DOI":"10.1109\/SBST52555.2021.00011"},{"key":"9620_CR61","doi-asserted-by":"crossref","unstructured":"Pham, H.\u00a0V., Kim, M., Tan, L., Yu, Y., & Nagappan, N. (2021). Deviate: A deep learning variance testing framework. In 2021 36th IEEE\/ACM International Conference on Automated Software Engineering (ASE) (pp. 1286\u20131290). IEEE.","DOI":"10.1109\/ASE51524.2021.9678540"},{"key":"9620_CR62","doi-asserted-by":"crossref","unstructured":"Pham, V.-T., B\u00f6hme, M., & Roychoudhury, A. (2020). Aflnet: a greybox fuzzer for network protocols. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST) (pp. 460\u2013465). IEEE.","DOI":"10.1109\/ICST46399.2020.00062"},{"key":"9620_CR63","doi-asserted-by":"crossref","unstructured":"Rapos, E.\u00a0J., & Cordy, J.\u00a0R. (2018). Simevo: A toolset for simulink test evolution & maintenance. In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST) (pp. 410\u2013415). IEEE.","DOI":"10.1109\/ICST.2018.00049"},{"key":"9620_CR64","doi-asserted-by":"crossref","unstructured":"Ren, M., Ma, F., Yin, Z., Li, H., Fu, Y., Chen, T., & Jiang, Y. (2021). Scstudio: a secure and efficient integrated development environment for smart contracts. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis (pp. 666\u2013669).","DOI":"10.1145\/3460319.3469078"},{"key":"9620_CR65","doi-asserted-by":"crossref","unstructured":"Ribeiro, H.\u00a0L., de\u00a0Araujo, R.\u00a0P., Chaim, M.\u00a0L., de\u00a0Souza, H.\u00a0A., & Kon, F. (2018). Jaguar: A spectrum-based fault localization tool for real-world software. In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST) (pp. 404\u2013409). IEEE.","DOI":"10.1109\/ICST.2018.00048"},{"key":"9620_CR66","doi-asserted-by":"crossref","unstructured":"Rizzi, E.\u00a0F., Elbaum, S., & Dwyer, M.\u00a0B. (2016). On the techniques we create, the tools we build, and their misalignments: A study of klee. In 2016 IEEE\/ACM 38th International Conference on Software Engineering (ICSE) (pp. 132\u2013143). IEEE.","DOI":"10.1145\/2884781.2884835"},{"key":"9620_CR67","doi-asserted-by":"crossref","unstructured":"Romdhana, A., Ceccato, M., Georgiu, G.\u00a0C., Merlo, A., & Tonella, P. (2021). Cosmo: code coverage made easier for android. In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST) (pp. 417\u2013423). IEEE.","DOI":"10.1109\/ICST49551.2021.00053"},{"key":"9620_CR68","doi-asserted-by":"crossref","unstructured":"Sahin, O., & Akay, B. (2021). A discrete dynamic artificial bee colony with hyper-scout for restful web service api test suite generation. Applied Soft Computing, 104, 107246.","DOI":"10.1016\/j.asoc.2021.107246"},{"key":"9620_CR69","doi-asserted-by":"crossref","unstructured":"Sartaj, H., Iqbal, M.\u00a0Z., & Khan, M.\u00a0U. (2020). Cdst: A toolkit for testing cockpit display systems. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST) (pp. 436\u2013441). IEEE.","DOI":"10.1109\/ICST46399.2020.00058"},{"key":"#cr-split#-9620_CR70.1","unstructured":"Slob, F.\u00a0J., & Jongmans, S.-S. (2021). Prut4j: Protocol unit testing fo"},{"key":"#cr-split#-9620_CR70.2","unstructured":"(u) r java. In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST) (pp. 448-453). IEEE."},{"key":"9620_CR71","doi-asserted-by":"crossref","unstructured":"Stallenberg, D., Olsthoorn, M., & Panichella, A. (2021). Improving test case generation for rest apis through hierarchical clustering. arXiv preprint arXiv:2109.06655","DOI":"10.1109\/ASE51524.2021.9678586"},{"key":"9620_CR72","doi-asserted-by":"crossref","unstructured":"Sullivan, A., Wang, K., & Khurshid, S. (2018). Aunit: A test automation tool for alloy. In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST) (pp. 398\u2013403). IEEE.","DOI":"10.1109\/ICST.2018.00047"},{"key":"9620_CR73","doi-asserted-by":"crossref","unstructured":"Tzoref-Brill, R., Sinha, S., Nassar, A.\u00a0A., Goldin, V., & Kermany, H. (2022). Tackletest: A tool for amplifying test generation via type-based combinatorial coverage. In 2022 15th IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE.","DOI":"10.1109\/ICST53961.2022.00050"},{"key":"9620_CR74","doi-asserted-by":"crossref","unstructured":"Udeshi, S., Jiang, X., & Chattopadhyay, S. (2020). Callisto: Entropy-based test generation and data quality assessment for machine learning systems. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST) (pp. 448\u2013453). IEEE.","DOI":"10.1109\/ICST46399.2020.00060"},{"key":"9620_CR75","doi-asserted-by":"crossref","unstructured":"Vadlamani, A., Kalicheti, R., & Chimalakonda, S. (2021). Apiscanner-towards automated detection of deprecated apis in python libraries. In 2021 IEEE\/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (pp. 5\u20138). IEEE.","DOI":"10.1109\/ICSE-Companion52605.2021.00022"},{"key":"9620_CR76","doi-asserted-by":"crossref","unstructured":"Viglianisi, E., Dallago, M., & Ceccato, M. (2020). Resttestgen: Automated black-box testing of restful apis. In IEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE.","DOI":"10.1109\/ICST46399.2020.00024"},{"key":"9620_CR77","doi-asserted-by":"crossref","unstructured":"Wang, B., Liu, H., Liu, C., Yang, Z., Ren, Q., Zheng, H., & Lei, H. (2021). Blockeye: Hunting for defi attacks on blockchain. In 2021 IEEE\/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (pp. 17\u201320). IEEE.","DOI":"10.1109\/ICSE-Companion52605.2021.00025"},{"key":"9620_CR78","doi-asserted-by":"crossref","unstructured":"Weiss, M., & Tonella, P. (2021). Uncertainty-wizard: Fast and user-friendly neural network uncertainty quantification. In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST) (pp. 436\u2013441). IEEE.","DOI":"10.1109\/ICST49551.2021.00056"},{"key":"9620_CR79","doi-asserted-by":"crossref","unstructured":"Xie, S., Yang, J., & Lu, S. (2021). Automated code refactoring upon database-schema changes in web applications. In 2021 36th IEEE\/ACM International Conference on Automated Software Engineering (ASE) (pp. 1262\u20131265). IEEE.","DOI":"10.1109\/ASE51524.2021.9678934"},{"key":"9620_CR80","doi-asserted-by":"crossref","unstructured":"Yavuz, T. (2022). Sift: A tool for property directed symbolic execution of multithreaded software. In 2022 15th IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE.","DOI":"10.1109\/ICST53961.2022.00049"},{"key":"9620_CR81","doi-asserted-by":"crossref","unstructured":"Zhang, M., & Arcuri, A. (2021a). Adaptive hypermutation for search-based system test generation: A study on rest apis with evomaster. ACM Transactions on Software Engineering and Methodology (TOSEM), 31.","DOI":"10.1145\/3464940"},{"key":"9620_CR82","doi-asserted-by":"crossref","unstructured":"Zhang, M., & Arcuri, A. (2021b). Enhancing resource-based test case generation for restful apis with sql handling. In International Symposium on Search Based Software Engineering (pp. 103\u2013117). Springer.","DOI":"10.1007\/978-3-030-88106-1_8"},{"key":"9620_CR83","doi-asserted-by":"crossref","unstructured":"Zhang, M., & Arcuri, A. (2022). Open problems in fuzzing restful apis: A comparison of tools. arXiv preprint arXiv:2205.05325","DOI":"10.1145\/3597205"},{"key":"9620_CR84","doi-asserted-by":"crossref","unstructured":"Zhang, M., Belhadi, A., & Arcuri, A. (2022). Javascript instrumentation for search-based software testing: A study with restful apis. In IEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE.","DOI":"10.1109\/ICST53961.2022.00022"},{"key":"9620_CR85","doi-asserted-by":"crossref","unstructured":"Zhang, M., Marculescu, B., & Arcuri, A. (2019). Resource-based test case generation for restful web services. In Proceedings of the Genetic and Evolutionary Computation Conference (pp. 1426\u20131434).","DOI":"10.1145\/3321707.3321815"},{"key":"9620_CR86","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10664-020-09937-1","volume":"26","author":"M Zhang","year":"2021","unstructured":"Zhang, M., Marculescu, B., & Arcuri, A. (2021). Resource and dependency based test case generation for restful web services. Empirical Software Engineering, 26, 1\u201361.","journal-title":"Empirical Software Engineering"}],"container-title":["Software Quality Journal"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11219-023-09620-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11219-023-09620-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11219-023-09620-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,31]],"date-time":"2023-08-31T10:26:30Z","timestamp":1693477590000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11219-023-09620-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,6]]},"references-count":87,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,9]]}},"alternative-id":["9620"],"URL":"https:\/\/doi.org\/10.1007\/s11219-023-09620-w","relation":{},"ISSN":["0963-9314","1573-1367"],"issn-type":[{"value":"0963-9314","type":"print"},{"value":"1573-1367","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,6]]},"assertion":[{"value":"7 February 2023","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 March 2023","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}]}}