{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,16]],"date-time":"2026-04-16T21:05:59Z","timestamp":1776373559975,"version":"3.51.2"},"publisher-location":"Cham","reference-count":43,"publisher":"Springer International Publishing","isbn-type":[{"value":"9783030714994","type":"print"},{"value":"9783030715007","type":"electronic"}],"license":[{"start":{"date-parts":[[2021,1,1]],"date-time":"2021-01-01T00:00:00Z","timestamp":1609459200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,3,20]],"date-time":"2021-03-20T00:00:00Z","timestamp":1616198400000},"content-version":"vor","delay-in-days":78,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>This report describes Test-Comp 2021, the 3rd edition of the Competition on Software Testing. The competition is a series of annual comparative evaluations of fully automatic software test generators for C programs. The competition has a strong focus on reproducibility of its results and its main goal is to provide an overview of the current state of the art in the area of automatic test-generation. The competition was based on 3\u00a0173 test-generation tasks for C\u00a0programs. Each test-generation task consisted of a program and a test specification (error coverage, branch coverage). Test-Comp 2021 had 11\u00a0participating test generators from 6\u00a0countries.<\/jats:p>","DOI":"10.1007\/978-3-030-71500-7_17","type":"book-chapter","created":{"date-parts":[[2021,3,19]],"date-time":"2021-03-19T13:12:14Z","timestamp":1616159534000},"page":"341-357","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":17,"title":["Status Report on Software Testing: Test-Comp 2021"],"prefix":"10.1007","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4832-7662","authenticated-orcid":false,"given":"Dirk","family":"Beyer","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,3,20]]},"reference":[{"key":"17_CR1","doi-asserted-by":"crossref","unstructured":"Alshmrany, K., Menezes, R., Gadelha, M., Cordeiro, L.: FuSeBMC: A white-box fuzzer for finding security vulnerabilities in C programs (competition contribution). In: Proc. FASE. LNCS 12649, Springer (2021)","DOI":"10.1007\/978-3-030-71500-7_19"},{"key":"17_CR2","doi-asserted-by":"publisher","unstructured":"Bartocci, E., Beyer, D., Black, P.E., Fedyukovich, G., Garavel, H., Hartmanns, A., Huisman, M., Kordon, F., Nagele, J., Sighireanu, M., Steffen, B., Suda, M., Sutcliffe, G., Weber, T., Yamada, A.: TOOLympics 2019: An overview of competitions in formal methods. In: Proc. TACAS (3). pp. 3\u201324. LNCS 11429, Springer (2019). https:\/\/doi.org\/10.1007\/978-3-030-17502-3_1","DOI":"10.1007\/978-3-030-17502-3_1"},{"key":"17_CR3","doi-asserted-by":"publisher","unstructured":"Beyer, D.: Second competition on software verification (Summary of SV-COMP 2013). In: Proc. TACAS. pp. 594\u2013609. LNCS 7795, Springer (2013). https:\/\/doi.org\/10.1007\/978-3-642-36742-7_43","DOI":"10.1007\/978-3-642-36742-7_43"},{"key":"17_CR4","doi-asserted-by":"publisher","unstructured":"Beyer, D.: Competition on software testing (Test-Comp). In: Proc. TACAS (3). pp. 167\u2013175. LNCS 11429, Springer (2019). https:\/\/doi.org\/10.1007\/978-3-030-17502-3_11","DOI":"10.1007\/978-3-030-17502-3_11"},{"key":"17_CR5","doi-asserted-by":"publisher","unstructured":"Beyer, D.: Second competition on software testing: Test-Comp 2020. In: Proc. FASE. pp. 505\u2013519. LNCS 12076, Springer (2020). https:\/\/doi.org\/10.1007\/978-3-030-45234-6_25","DOI":"10.1007\/978-3-030-45234-6_25"},{"key":"17_CR6","doi-asserted-by":"crossref","unstructured":"Beyer, D.: First international competition on software testing (Test-Comp 2019). Int. J. Softw. Tools Technol. Transf. (2021)","DOI":"10.1007\/s10009-021-00611-5"},{"key":"17_CR7","doi-asserted-by":"publisher","unstructured":"Beyer, D.: Results of the 3rd Intl. Competition on Software Testing (Test-Comp 2021). Zenodo (2021). https:\/\/doi.org\/10.5281\/zenodo.4459470","DOI":"10.5281\/zenodo.4459470"},{"key":"17_CR8","doi-asserted-by":"crossref","unstructured":"Beyer, D.: Software verification: 10th comparative evaluation (SV-COMP 2021). In: Proc. TACAS\u00a0(2). LNCS\u00a012652, Springer (2021), preprint available","DOI":"10.1007\/978-3-030-72013-1_24"},{"key":"17_CR9","doi-asserted-by":"publisher","unstructured":"Beyer, D.: SV-Benchmarks: Benchmark set of 3rd Intl. Competition on Software Testing (Test-Comp 2021). Zenodo (2021). https:\/\/doi.org\/10.5281\/zenodo.4459132","DOI":"10.5281\/zenodo.4459132"},{"key":"17_CR10","doi-asserted-by":"publisher","unstructured":"Beyer, D.: Test suites from Test-Comp 2021 test-generation tools. Zenodo (2021). https:\/\/doi.org\/10.5281\/zenodo.4459466","DOI":"10.5281\/zenodo.4459466"},{"key":"17_CR11","doi-asserted-by":"publisher","unstructured":"Beyer, D., Chlipala, A.J., Henzinger, T.A., Jhala, R., Majumdar, R.: Generating tests from counterexamples. In: Proc. ICSE. pp. 326\u2013335. IEEE (2004). https:\/\/doi.org\/10.1109\/ICSE.2004.1317455","DOI":"10.1109\/ICSE.2004.1317455"},{"key":"17_CR12","doi-asserted-by":"publisher","unstructured":"Beyer, D., Jakobs, M.C.: CoVeriTest: Cooperative verifier-based testing. In: Proc. FASE. pp. 389\u2013408. LNCS 11424, Springer (2019). https:\/\/doi.org\/10.1007\/978-3-030-16722-6_23","DOI":"10.1007\/978-3-030-16722-6_23"},{"key":"17_CR13","doi-asserted-by":"crossref","unstructured":"Beyer, D., Kanav, S.: CoVeriTeam: On-demand composition of cooperative verification systems. unpublished manuscript (2021)","DOI":"10.1007\/978-3-030-99524-9_31"},{"key":"17_CR14","doi-asserted-by":"publisher","unstructured":"Beyer, D., Lemberger, T.: Software verification: Testing vs. model checking. In: Proc. HVC. pp. 99\u2013114. LNCS 10629, Springer (2017). https:\/\/doi.org\/10.1007\/978-3-319-70389-3_7","DOI":"10.1007\/978-3-319-70389-3_7"},{"key":"17_CR15","doi-asserted-by":"publisher","unstructured":"Beyer, D., Lemberger, T.: TestCov: Robust test-suite execution and coverage measurement. In: Proc. ASE. pp. 1074\u20131077. IEEE (2019). https:\/\/doi.org\/10.1109\/ASE.2019.00105","DOI":"10.1109\/ASE.2019.00105"},{"key":"17_CR16","doi-asserted-by":"publisher","unstructured":"Beyer, D., L\u00f6we, S., Wendler, P.: Reliable benchmarking: Requirements and solutions. Int. J. Softw. Tools Technol. Transfer 21(1), 1\u201329 (2019). https:\/\/doi.org\/10.1007\/s10009-017-0469-y","DOI":"10.1007\/s10009-017-0469-y"},{"key":"17_CR17","doi-asserted-by":"publisher","unstructured":"Beyer, D., Wendler, P.: CPU CPU Energy Meter: A tool for energy-aware algorithms engineering. In: Proc. TACAS (2). pp. 126\u2013133. LNCS 12079, Springer (2020). https:\/\/doi.org\/10.1007\/978-3-030-45237-7_8","DOI":"10.1007\/978-3-030-45237-7_8"},{"key":"17_CR18","doi-asserted-by":"publisher","unstructured":"B\u00fcrdek, J., Lochau, M., Bauregger, S., Holzer, A., von Rhein, A., Apel, S., Beyer, D.: Facilitating reuse in multi-goal test-suite generation for software product lines. In: Proc. FASE. pp. 84\u201399. LNCS 9033, Springer (2015). https:\/\/doi.org\/10.1007\/978-3-662-46675-9_6","DOI":"10.1007\/978-3-662-46675-9_6"},{"key":"17_CR19","unstructured":"Cadar, C., Dunbar, D., Engler, D.R.: Klee: Unassisted and automatic generation of high-coverage tests for complex systems programs. In: Proc. OSDI. pp. 209\u2013224. USENIX Association (2008)"},{"key":"17_CR20","doi-asserted-by":"publisher","unstructured":"Cadar, C., Nowack, M.: Klee symbolic execution engine in 2019. Int. J. Softw. Tools Technol. Transf. (2020). https:\/\/doi.org\/10.1007\/s10009-020-00570-3","DOI":"10.1007\/s10009-020-00570-3"},{"key":"17_CR21","doi-asserted-by":"crossref","unstructured":"Chalupa, M., Nov\u00e1k, J., Strej\u010dek, J.: Symbiotic 8: Parallel and targeted test generation (competition contribution). In: Proc. FASE. LNCS 12649, Springer (2021)","DOI":"10.1007\/978-3-030-71500-7_20"},{"key":"17_CR22","doi-asserted-by":"publisher","unstructured":"Chalupa, M., Strej\u010dek, J., Vitovsk\u00e1, M.: Joint forces for memory safety checking. In: Proc. SPIN. pp. 115\u2013132. Springer (2018). https:\/\/doi.org\/10.1007\/978-3-319-94111-0_7","DOI":"10.1007\/978-3-319-94111-0_7"},{"key":"17_CR23","doi-asserted-by":"publisher","unstructured":"Chowdhury, A.B., Medicherla, R.K., Venkatesh, R.: VeriFuzz: Program-aware fuzzing (competition contribution). In: Proc. TACAS (3). pp. 244\u2013249. LNCS 11429, Springer (2019). https:\/\/doi.org\/10.1007\/978-3-030-17502-3_22","DOI":"10.1007\/978-3-030-17502-3_22"},{"key":"17_CR24","doi-asserted-by":"crossref","unstructured":"Cok, D.R., D\u00e9harbe, D., Weber, T.: The 2014 SMT competition. JSAT 9, 207\u2013242 (2016)","DOI":"10.3233\/SAT190109"},{"key":"17_CR25","doi-asserted-by":"publisher","unstructured":"Gadelha, M.R., Menezes, R., Cordeiro, L.: Esbmc 6.1: Automated test-case generation using bounded model checking. Int. J. Softw. Tools Technol. Transf. (2020). https:\/\/doi.org\/10.1007\/s10009-020-00571-2","DOI":"10.1007\/s10009-020-00571-2"},{"key":"17_CR26","doi-asserted-by":"publisher","unstructured":"Godefroid, P., Sen, K.: Combining model checking and testing. In: Handbook of Model Checking, pp. 613\u2013649. Springer (2018). https:\/\/doi.org\/10.1007\/978-3-319-10575-8_19","DOI":"10.1007\/978-3-319-10575-8_19"},{"key":"17_CR27","doi-asserted-by":"publisher","unstructured":"Harman, M., Hu, L., Hierons, R.M., Wegener, J., Sthamer, H., Baresel, A., Roper, M.: Testability transformation. IEEE Trans. Software Eng. 30(1), 3\u201316 (2004). https:\/\/doi.org\/10.1109\/TSE.2004.1265732","DOI":"10.1109\/TSE.2004.1265732"},{"key":"17_CR28","doi-asserted-by":"publisher","unstructured":"Holzer, A., Schallhart, C., Tautschnig, M., Veith, H.: How did you specify your test suite. In: Proc. ASE. pp. 407\u2013416. ACM (2010). https:\/\/doi.org\/10.1145\/1858996.1859084","DOI":"10.1145\/1858996.1859084"},{"key":"17_CR29","doi-asserted-by":"publisher","unstructured":"Jaffar, J., Maghareh, R., Godboley, S., Ha, X.L.: TracerX: Dynamic symbolic execution with interpolation (competition contribution). In: Proc. FASE. pp. 530\u2013534. LNCS 12076, Springer (2020). https:\/\/doi.org\/10.1007\/978-3-030-45234-6_28","DOI":"10.1007\/978-3-030-45234-6_28"},{"key":"17_CR30","doi-asserted-by":"publisher","unstructured":"Jaffar, J., Murali, V., Navas, J.A., Santosa, A.E.: Tracer: A symbolic execution tool for verification. In: Proc. CAV. pp. 758\u2013766. LNCS 7358, Springer (2012). https:\/\/doi.org\/10.1007\/978-3-642-31424-7_61","DOI":"10.1007\/978-3-642-31424-7_61"},{"key":"17_CR31","doi-asserted-by":"crossref","unstructured":"Jakobs, M.C., Richter, C.: CoVeriTest with adaptive time scheduling (competition contribution). In: Proc. FASE. LNCS 12649, Springer (2021)","DOI":"10.1007\/978-3-030-71500-7_18"},{"key":"17_CR32","doi-asserted-by":"publisher","unstructured":"Kifetew, F.M., Devroey, X., Rueda, U.: Java unit-testing tool competition: Seventh round. In: Proc. SBST. pp. 15\u201320. IEEE (2019). https:\/\/doi.org\/10.1109\/SBST.2019.00014","DOI":"10.1109\/SBST.2019.00014"},{"key":"17_CR33","unstructured":"Kim, H.: Fuzzing with stochastic optimization (2020), Bachelor\u2019s Thesis, LMU Munich"},{"key":"17_CR34","doi-asserted-by":"publisher","unstructured":"King, J.C.: Symbolic execution and program testing. Commun. ACM 19(7), 385\u2013394 (1976). https:\/\/doi.org\/10.1145\/360248.360252","DOI":"10.1145\/360248.360252"},{"key":"17_CR35","doi-asserted-by":"publisher","unstructured":"Le, H.M.: Llvm-based hybrid fuzzing with LibKluzzer (competition contribution). In: Proc. FASE. pp. 535\u2013539. LNCS 12076, Springer (2020). https:\/\/doi.org\/10.1007\/978-3-030-45234-6_29","DOI":"10.1007\/978-3-030-45234-6_29"},{"key":"17_CR36","doi-asserted-by":"crossref","unstructured":"Lemberger, T.: Plain random test generation with PRTest. Int. J. Softw. Tools Technol. Transf. (2020)","DOI":"10.1007\/s10009-020-00568-x"},{"key":"17_CR37","doi-asserted-by":"publisher","unstructured":"Liu, D., Ernst, G., Murray, T., Rubinstein, B.: Legion: Best-first concolic testing (competition contribution). In: Proc. FASE. pp. 545\u2013549. LNCS 12076, Springer (2020). https:\/\/doi.org\/10.1007\/978-3-030-45234-6_31","DOI":"10.1007\/978-3-030-45234-6_31"},{"key":"17_CR38","doi-asserted-by":"publisher","unstructured":"Ruland, S., Lochau, M., Jakobs, M.C.: HybridTiger: Hybrid model checking and domination-based partitioning for efficient multi-goal test-suite generation (competition contribution). In: Proc. FASE. pp. 520\u2013524. LNCS 12076, Springer (2020). https:\/\/doi.org\/10.1007\/978-3-030-45234-6_26","DOI":"10.1007\/978-3-030-45234-6_26"},{"key":"17_CR39","doi-asserted-by":"publisher","unstructured":"Song, J., Alves-Foss, J.: The DARPA cyber grand challenge: A competitor\u2019s perspective, part 2. IEEE Security and Privacy 14(1), 76\u201381 (2016). https:\/\/doi.org\/10.1109\/MSP.2016.14","DOI":"10.1109\/MSP.2016.14"},{"key":"17_CR40","doi-asserted-by":"publisher","unstructured":"Stump, A., Sutcliffe, G., Tinelli, C.: StarExec: A cross-community infrastructure for logic solving. In: Proc. IJCAR, pp. 367\u2013373. LNCS 8562, Springer (2014). https:\/\/doi.org\/10.1007\/978-3-319-08587-6_28","DOI":"10.1007\/978-3-319-08587-6_28"},{"key":"17_CR41","doi-asserted-by":"crossref","unstructured":"Sutcliffe, G.: The CADE ATP system competition: CASC. AI Magazine 37(2), 99\u2013101 (2016)","DOI":"10.1609\/aimag.v37i2.2620"},{"key":"17_CR42","doi-asserted-by":"publisher","unstructured":"Visser, W., P\u0103s\u0103reanu, C.S., Khurshid, S.: Test-input generation with Java PathFinder. In: Proc. ISSTA. pp. 97\u2013107. ACM (2004). https:\/\/doi.org\/10.1145\/1007512.1007526","DOI":"10.1145\/1007512.1007526"},{"key":"17_CR43","doi-asserted-by":"publisher","unstructured":"Wendler, P., Beyer, D.: sosy-lab\/benchexec: Release 3.6. Zenodo (2021). https:\/\/doi.org\/10.5281\/zenodo.4317433","DOI":"10.5281\/zenodo.4317433"}],"container-title":["Lecture Notes in Computer Science","Fundamental Approaches to Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-030-71500-7_17","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,22]],"date-time":"2022-12-22T00:02:53Z","timestamp":1671667373000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/978-3-030-71500-7_17"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021]]},"ISBN":["9783030714994","9783030715007"],"references-count":43,"URL":"https:\/\/doi.org\/10.1007\/978-3-030-71500-7_17","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"value":"0302-9743","type":"print"},{"value":"1611-3349","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021]]},"assertion":[{"value":"20 March 2021","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"FASE","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"International Conference on Fundamental Approaches to Software Engineering","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Luxembourg City","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Luxembourg","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2021","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"27 March 2021","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"1 April 2021","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"24","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"fase2021","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/etaps.org\/2021\/fase","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Double-blind","order":1,"name":"type","label":"Type","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"EasyChair","order":2,"name":"conference_management_system","label":"Conference Management System","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"52","order":3,"name":"number_of_submissions_sent_for_review","label":"Number of Submissions Sent for Review","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"16","order":4,"name":"number_of_full_papers_accepted","label":"Number of Full Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"0","order":5,"name":"number_of_short_papers_accepted","label":"Number of Short Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"31% - The value is computed by the equation \"Number of Full Papers Accepted \/ Number of Submissions Sent for Review * 100\" and then rounded to a whole number.","order":6,"name":"acceptance_rate_of_full_papers","label":"Acceptance Rate of Full Papers","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"3","order":7,"name":"average_number_of_reviews_per_paper","label":"Average Number of Reviews per Paper","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"5,5","order":8,"name":"average_number_of_papers_per_reviewer","label":"Average Number of Papers per Reviewer","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"Yes","order":9,"name":"external_reviewers_involved","label":"External Reviewers Involved","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"The conference changed to an online format due to the COVID-19 pandemic.","order":10,"name":"additional_info_on_review_process","label":"Additional Info on Review Process","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}}]}}