{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,5]],"date-time":"2025-08-05T12:35:45Z","timestamp":1754397345577,"version":"3.40.3"},"publisher-location":"Cham","reference-count":43,"publisher":"Springer International Publishing","isbn-type":[{"type":"print","value":"9783030720186"},{"type":"electronic","value":"9783030720193"}],"license":[{"start":{"date-parts":[[2021,1,1]],"date-time":"2021-01-01T00:00:00Z","timestamp":1609459200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,3,23]],"date-time":"2021-03-23T00:00:00Z","timestamp":1616457600000},"content-version":"vor","delay-in-days":81,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p><jats:italic>Property-based testing<\/jats:italic> uses randomly generated inputs to validate high-level program specifications. It can be shockingly effective at finding bugs, but it often requires generating a very large number of inputs to do so. In this paper, we apply ideas from <jats:italic>combinatorial testing<\/jats:italic>, a powerful and widely studied testing methodology, to modify the distributions of our random generators so as to find bugs with fewer tests. The key concept is <jats:italic>combinatorial coverage<\/jats:italic>, which measures the degree to which a given set of tests exercises every possible choice of values for every small combination of input features.<\/jats:p><jats:p>In its \u201cclassical\u201d form, combinatorial coverage only applies to programs whose inputs have a very particular shape\u2014essentially, a Cartesian product of finite sets. We generalize combinatorial coverage to the richer world of algebraic data types by formalizing a class of <jats:italic>sparse test descriptions<\/jats:italic> based on regular tree expressions. This new definition of coverage inspires a novel <jats:italic>combinatorial thinning<\/jats:italic> algorithm for improving the coverage of random test generators, requiring many fewer tests to catch bugs. We evaluate this algorithm on two case studies, a typed evaluator for System F terms and a Haskell compiler, showing significant improvements in both.<\/jats:p>","DOI":"10.1007\/978-3-030-72019-3_10","type":"book-chapter","created":{"date-parts":[[2021,3,22]],"date-time":"2021-03-22T14:03:10Z","timestamp":1616421790000},"page":"264-291","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Do Judge a Test by its Cover"],"prefix":"10.1007","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9631-1169","authenticated-orcid":false,"given":"Harrison","family":"Goldstein","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8042-0969","authenticated-orcid":false,"given":"John","family":"Hughes","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0269-9815","authenticated-orcid":false,"given":"Leonidas","family":"Lampropoulos","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7839-1636","authenticated-orcid":false,"given":"Benjamin C.","family":"Pierce","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,3,23]]},"reference":[{"key":"10_CR1","doi-asserted-by":"publisher","unstructured":"Arcuri, A., Briand, L.C.: Adaptive random testing: an illusion of effectiveness? In: Dwyer, M.B., Tip, F. (eds.) Proceedings of the 20th International Symposium on Software Testing and Analysis, ISSTA 2011,Toronto, ON, Canada, July 17-21, 2011. pp. 265\u2013275. ACM (2011). https:\/\/doi.org\/10.1145\/2001420.2001452, https:\/\/doi.org\/10.1145\/2001420.2001452","DOI":"10.1145\/2001420.2001452"},{"key":"10_CR2","unstructured":"Bell, K.Z., Vouk, M.A.: On effectiveness of pairwise methodology for testing network-centric software. In: 2005 International Conference on Information and Communication Technology. pp. 221\u2013235. IEEE (2005)"},{"key":"10_CR3","unstructured":"Braquehais, R.M.: Tools for discovery, refinement and generalization of functional properties by enumerative testing (October 2017), http:\/\/etheses.whiterose.ac.uk\/19178\/"},{"key":"10_CR4","doi-asserted-by":"publisher","unstructured":"Chen, T.Y., Leung, H., Mak, I.K.: Adaptive random testing. In: Maher, M.J. (ed.) Advances in Computer Science - ASIAN 2004, Higher-Level Decision Making, 9th Asian Computing Science Conference, Dedicated to Jean-Louis Lassez on the Occasion of His 5th Cycle Birthday, Chiang Mai, Thailand, December 8-10, 2004, Proceedings. Lecture Notes in Computer Science, vol.\u00a03321, pp. 320\u2013329. Springer (2004). https:\/\/doi.org\/10.1007\/978-3-540-30502-6_23, https:\/\/doi.org\/10.1007\/978-3-540-30502-6_23","DOI":"10.1007\/978-3-540-30502-6_23"},{"key":"10_CR5","doi-asserted-by":"publisher","unstructured":"Ciupa, I., Leitner, A., Oriol, M., Meyer, B.: ARTOO: adaptive random testing for object-oriented software. In: Sch\u00e4fer, W., Dwyer, M.B., Gruhn, V. (eds.) 30th International Conference on Software Engineering (ICSE 2008), Leipzig, Germany, May 10-18, 2008. pp. 71\u201380. ACM (2008). https:\/\/doi.org\/10.1145\/1368088.1368099, https:\/\/doi.org\/10.1145\/1368088.1368099","DOI":"10.1145\/1368088.1368099"},{"key":"10_CR6","doi-asserted-by":"publisher","unstructured":"Claessen, K., Dureg\u00e5rd, J., Palka, M.H.: Generating constrained random data with uniform distribution. J. Funct. Program. 25 (2015). https:\/\/doi.org\/10.1017\/S0956796815000143, http:\/\/dx.doi.org\/10.1017\/S0956796815000143","DOI":"10.1017\/S0956796815000143"},{"key":"10_CR7","doi-asserted-by":"publisher","unstructured":"Claessen, K., Hughes, J.: Quickcheck: a lightweight tool for random testing of haskell programs. In: Odersky, M., Wadler, P. (eds.) Proceedings of the Fifth ACM SIGPLAN International Conference on Functional Programming (ICFP \u201900), Montreal, Canada, September 18-21, 2000. pp. 268\u2013279. ACM (2000). https:\/\/doi.org\/10.1145\/351240.351266, https:\/\/doi.org\/10.1145\/351240.351266","DOI":"10.1145\/351240.351266"},{"key":"10_CR8","unstructured":"Colbourn, C.J., Cohen, M.B., Turban, R.: A deterministic density algorithm for pairwise interaction coverage. In: Hamza, M.H. (ed.) IASTED International Conference on Software Engineering, part of the 22nd Multi-Conference on Applied Informatics, Innsbruck, Austria, February 17-19, 2004. pp. 345\u2013352. IASTED\/ACTA Press (2004)"},{"key":"10_CR9","unstructured":"Comon, H., Dauchet, M., Gilleron, R., L\u00f6ding, C., Jacquemard, F., Lugiez, D., Tison, S., Tommasi, M.: Tree automata techniques and applications. Available on: http:\/\/www.grappa.univ-lille3.fr\/tata (2007), release October, 12th 2007"},{"key":"10_CR10","doi-asserted-by":"publisher","unstructured":"Courcelle, B.: Fundamental properties of infinite trees. Theor. Comput. Sci. 25, 95\u2013169 (1983). https:\/\/doi.org\/10.1016\/0304-3975(83)90059-2, https:\/\/doi.org\/10.1016\/0304-3975(83)90059-2","DOI":"10.1016\/0304-3975(83)90059-2"},{"key":"10_CR11","doi-asserted-by":"publisher","unstructured":"Gill, A., Runciman, C.: Haskell program coverage. In: Keller, G. (ed.) Proceedings of the ACM SIGPLAN Workshop on Haskell, Haskell 2007, Freiburg, Germany, September 30, 2007. pp. 1\u201312. ACM (2007). https:\/\/doi.org\/10.1145\/1291201.1291203, https:\/\/doi.org\/10.1145\/1291201.1291203","DOI":"10.1145\/1291201.1291203"},{"key":"10_CR12","unstructured":"Girard, J.Y.: Interpr\u00e9tation fonctionnelle et \u00e9limination des coupuresde l\u2019arithm\u00e9tique d\u2019ordre sup\u00e9rieur. Ph.D. thesis, \u00c9diteur inconnu (1972)"},{"key":"10_CR13","doi-asserted-by":"publisher","unstructured":"Havrikov, N., Zeller, A.: Systematically covering input structure. In: 34th IEEE\/ACM International Conference on Automated Software Engineering, ASE 2019, San Diego, CA, USA, November 11-15, 2019. pp. 189\u2013199. IEEE (2019). https:\/\/doi.org\/10.1109\/ASE.2019.00027, https:\/\/doi.org\/10.1109\/ASE.2019.00027","DOI":"10.1109\/ASE.2019.00027"},{"key":"10_CR14","doi-asserted-by":"publisher","unstructured":"Kuhn, D.R., Wallace, D.R., Gallo, A.M.: Software fault interactions and implications for software testing. IEEE Transactions on Software Engineering 30(6), 418\u2013421 (2004). https:\/\/doi.org\/10.1109\/TSE.2004.24","DOI":"10.1109\/TSE.2004.24"},{"key":"10_CR15","doi-asserted-by":"publisher","unstructured":"Kuhn, D.R., Higdon, J.M., Lawrence, J., Kacker, R., Lei, Y.: Combinatorial methods for event sequence testing. In: Antoniol, G., Bertolino, A., Labiche, Y. (eds.) Fifth IEEE International Conference on Software Testing, Verification and Validation, ICST 2012, Montreal, QC, Canada, April 17-21, 2012. pp. 601\u2013609. IEEE Computer Society (2012). https:\/\/doi.org\/10.1109\/ICST.2012.147, https:\/\/doi.org\/10.1109\/ICST.2012.147","DOI":"10.1109\/ICST.2012.147"},{"key":"10_CR16","doi-asserted-by":"crossref","unstructured":"Kuhn, D.R., Kacker, R.N., Lei, Y.: Practical combinatorial testing. NIST special Publication 800(142), \u00a0142 (2010)","DOI":"10.6028\/NIST.SP.800-142"},{"key":"10_CR17","doi-asserted-by":"crossref","unstructured":"Kuhn, D.R., Wallace, D.R., Gallo, A.M.: Software fault interactions and implications for software testing. IEEE transactions on software engineering 30(6), 418\u2013421 (2004)","DOI":"10.1109\/TSE.2004.24"},{"key":"10_CR18","doi-asserted-by":"publisher","unstructured":"Kuraj, I., Kuncak, V., Jackson, D.: Programming with enumerable sets of structures. In: Aldrich, J., Eugster, P. (eds.) Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2015, part of SPLASH 2015, Pittsburgh, PA, USA, October 25-30, 2015. pp. 37\u201356. ACM (2015). https:\/\/doi.org\/10.1145\/2814270.2814323, https:\/\/doi.org\/10.1145\/2814270.2814323","DOI":"10.1145\/2814270.2814323"},{"key":"10_CR19","doi-asserted-by":"publisher","unstructured":"L\u00e4mmel, R., Schulte, W.: Controllable combinatorial coverage in grammar-based testing. In: Uyar, M.\u00dc., Duale, A.Y., Fecko, M.A. (eds.) Testing of Communicating Systems, 18th IFIP TC6\/WG6.1 International Conference, TestCom 2006, New York, NY, USA, May 16-18, 2006, Proceedings. Lecture Notes in Computer Science, vol.\u00a03964, pp. 19\u201338. Springer (2006). https:\/\/doi.org\/10.1007\/11754008_2, https:\/\/doi.org\/10.1007\/11754008_2","DOI":"10.1007\/11754008_2"},{"key":"10_CR20","doi-asserted-by":"crossref","unstructured":"Lampropoulos, L., Gallois-Wong, D., Hritcu, C., Hughes, J., Pierce, B.C., Xia, L.: Beginner\u2019s Luck: a language for property-based generators. In: Proceedings of the 44th ACM SIGPLAN Symposium on Principles of Programming Languages, POPL 2017, Paris, France, January 18-20, 2017. pp. 114\u2013129 (2017), http:\/\/dl.acm.org\/citation.cfm?id=3009868","DOI":"10.1145\/3009837.3009868"},{"key":"10_CR21","doi-asserted-by":"publisher","unstructured":"Lampropoulos, L., Hicks, M., Pierce, B.C.: Coverage guided, property based testing. PACMPL 3(OOPSLA), 181:1\u2013181:29 (2019). https:\/\/doi.org\/10.1145\/3360607, https:\/\/doi.org\/10.1145\/3360607","DOI":"10.1145\/3360607"},{"key":"10_CR22","unstructured":"lcamtuf: AFL quick start guide. http:\/\/lcamtuf.coredump.cx\/afl\/QuickStartGuide.txt (Apr 2018)"},{"key":"10_CR23","doi-asserted-by":"crossref","unstructured":"Lei, Y., Kacker, R., Kuhn, D.R., Okun, V., Lawrence, J.: Ipog: A general strategy for t-way software testing. In: 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems (ECBS\u201907). pp. 549\u2013556. IEEE (2007)","DOI":"10.1109\/ECBS.2007.47"},{"key":"10_CR24","doi-asserted-by":"crossref","unstructured":"Livinskii, V., Babokin, D., Regehr, J.: Random testing for c and c++ compilers with yarpgen. Proceedings of the ACM on Programming Languages 4(OOPSLA), 1\u201325 (2020)","DOI":"10.1145\/3428264"},{"key":"10_CR25","doi-asserted-by":"crossref","unstructured":"Luo, Q., Hariri, F., Eloussi, L., Marinov, D.: An empirical analysis of flaky tests. In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. pp. 643\u2013653 (2014)","DOI":"10.1145\/2635868.2635920"},{"key":"10_CR26","doi-asserted-by":"publisher","unstructured":"Mariani, L., Pezz\u00e8, M., Willmor, D.: Generation of integration tests for self-testing components. In: N\u00fa\u00f1ez, M., Maamar, Z., Pelayo, F.L., Pousttchi, K., Rubio, F. (eds.) Applying Formal Methods: Testing, Performance and M\/ECommerce, FORTE 2004 Workshops The FormEMC, EPEW, ITM, Toledo, Spain, October 1-2, 2004. Lecture Notes in Computer Science, vol.\u00a03236, pp. 337\u2013350. Springer (2004). https:\/\/doi.org\/10.1007\/978-3-540-30233-9_25, https:\/\/doi.org\/10.1007\/978-3-540-30233-9_25","DOI":"10.1007\/978-3-540-30233-9_25"},{"key":"10_CR27","unstructured":"Marlow, S., Peyton-Jones, S.: The Glasgow Haskell Compiler. In: Brown, A., Wilson, G. (eds.) The Architecture of Open Source Applications, vol.\u00a0II. Available online under the Creative Commons Attribution 3.0 Unported license (March 2012), http:\/\/www.aosabook.org\/en\/ghc.html"},{"key":"10_CR28","doi-asserted-by":"publisher","unstructured":"Mathis, B., Gopinath, R., Zeller, A.: Learning input tokens for effective fuzzing. In: Khurshid, S., Pasareanu, C.S. (eds.) ISSTA \u201920: 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, USA, July 18-22, 2020. pp. 27\u201337. ACM (2020). https:\/\/doi.org\/10.1145\/3395363.3397348, https:\/\/doi.org\/10.1145\/3395363.3397348","DOI":"10.1145\/3395363.3397348"},{"key":"10_CR29","doi-asserted-by":"publisher","unstructured":"Mista, A., Russo, A., Hughes, J.: Branching processes for quickcheck generators. In: Wu, N. (ed.) Proceedings of the 11th ACM SIGPLAN International Symposium on Haskell, askell@ICFP 2018, St. Louis, MO, USA, September 27-17, 2018. pp. 1\u201313. ACM (2018). https:\/\/doi.org\/10.1145\/3242744.3242747, https:\/\/doi.org\/10.1145\/3242744.3242747","DOI":"10.1145\/3242744.3242747"},{"key":"10_CR30","doi-asserted-by":"publisher","unstructured":"Nie, C., Leung, H.: A survey of combinatorial testing. ACM Comput. Surv. 43(2) (Feb 2011). https:\/\/doi.org\/10.1145\/1883612.1883618, https:\/\/doi.org\/10.1145\/1883612.1883618","DOI":"10.1145\/1883612.1883618"},{"key":"10_CR31","doi-asserted-by":"publisher","unstructured":"Padhye, R., Lemieux, C., Sen, K., Simon, L., Vijayakumar, H.: Fuzzfactory: domain-specific fuzzing with waypoints. Proc. ACM Program. Lang. 3(OOPSLA), 174:1\u2013174:29 (2019). https:\/\/doi.org\/10.1145\/3360600, https:\/\/doi.org\/10.1145\/3360600","DOI":"10.1145\/3360600"},{"key":"10_CR32","doi-asserted-by":"publisher","unstructured":"Pa\u0142ka, M.H., Claessen, K., Russo, A., Hughes, J.: Testing an optimising compiler by generating random lambda terms. In: Proceedings of the 6th International Workshop on Automation of Software Test. pp. 91\u201397. AST \u201911, ACM, New York, NY, USA (2011). https:\/\/doi.org\/10.1145\/1982595.1982615, http:\/\/doi.acm.org\/10.1145\/1982595.1982615","DOI":"10.1145\/1982595.1982615"},{"key":"10_CR33","doi-asserted-by":"publisher","unstructured":"Reddy, S., Lemieux, C., Padhye, R., Sen, K.: Quickly generating diverse valid test inputs with reinforcement learning. In: Rothermel, G., Bae, D. (eds.) ICSE \u201920: 42nd International Conference on Software Engineering, Seoul, South Korea, 27 June - 19 July, 2020. pp. 1410\u20131421. ACM (2020). https:\/\/doi.org\/10.1145\/3377811.3380399, https:\/\/doi.org\/10.1145\/3377811.3380399","DOI":"10.1145\/3377811.3380399"},{"key":"10_CR34","doi-asserted-by":"publisher","unstructured":"Regehr, J., Chen, Y., Cuoq, P., Eide, E., Ellison, C., Yang, X.: Test-case reduction for C compiler bugs. In: ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI \u201912, Beijing, China - June 11 - 16, 2012. pp. 335\u2013346 (2012). https:\/\/doi.org\/10.1145\/2254064.2254104, http:\/\/www.cs.utah.edu\/~regehr\/papers\/pldi12-preprint.pdf","DOI":"10.1145\/2254064.2254104"},{"key":"10_CR35","doi-asserted-by":"publisher","unstructured":"Reynolds, J.C.: Towards a theory of type structure. In: Robinet, B. (ed.) Programming Symposium, Proceedings Colloque sur la Programmation, Paris, France, April 9-11, 1974. Lecture Notes in Computer Science, vol.\u00a019, pp. 408\u2013423. Springer (1974). https:\/\/doi.org\/10.1007\/3-540-06859-7_148, https:\/\/doi.org\/10.1007\/3-540-06859-7_148","DOI":"10.1007\/3-540-06859-7_148"},{"key":"10_CR36","doi-asserted-by":"publisher","unstructured":"Runciman, C., Naylor, M., Lindblad, F.: Smallcheck and lazy smallcheck: automatic exhaustive testing for small values. In: Gill, A. (ed.) Proceedings of the 1st ACM SIGPLAN Symposium on Haskell, Haskell 2008, Victoria, BC, Canada, 25 September 2008. pp. 37\u201348. ACM (2008). https:\/\/doi.org\/10.1145\/1411286.1411292, https:\/\/doi.org\/10.1145\/1411286.1411292","DOI":"10.1145\/1411286.1411292"},{"key":"10_CR37","doi-asserted-by":"publisher","unstructured":"Salecker, E., Glesner, S.: Combinatorial interaction testing for test selection in grammar-based testing. In: Antoniol, G., Bertolino, A., Labiche, Y. (eds.) Fifth IEEE International Conference on Software Testing, Verification and Validation, ICST 2012, Montreal, QC, Canada, April 17-21, 2012. pp. 610\u2013619. IEEE Computer Society (2012). https:\/\/doi.org\/10.1109\/ICST.2012.148, https:\/\/doi.org\/10.1109\/ICST.2012.148","DOI":"10.1109\/ICST.2012.148"},{"key":"10_CR38","doi-asserted-by":"publisher","unstructured":"Sarkar, K., Colbourn, C.J.: Upper bounds on the size of covering arrays. SIAM J. Discrete Math. 31(2), 1277\u20131293 (2017). https:\/\/doi.org\/10.1137\/16M1067767, https:\/\/doi.org\/10.1137\/16M1067767","DOI":"10.1137\/16M1067767"},{"key":"10_CR39","doi-asserted-by":"crossref","unstructured":"Stephens, N., Grosen, J., Salls, C., Dutcher, A., Wang, R., Corbetta, J., Shoshitaishvili, Y., Kruegel, C., Vigna, G.: Driller: Augmenting fuzzing through selective symbolic execution. In: Network and Distributed System Security Symposium (NDSS) (2016)","DOI":"10.14722\/ndss.2016.23368"},{"key":"10_CR40","doi-asserted-by":"publisher","unstructured":"Usaola, M.P., Romero, F.R., Aranda, R.R., de\u00a0Guzm\u00e1n, I.G.R.: Test case generation with regular expressions and combinatorial techniques. In: 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops, ICST Workshops 2017, Tokyo, Japan, March 13-17, 2017. pp. 189\u2013198. IEEE Computer Society (2017). https:\/\/doi.org\/10.1109\/ICSTW.2017.38, https:\/\/doi.org\/10.1109\/ICSTW.2017.38","DOI":"10.1109\/ICSTW.2017.38"},{"key":"10_CR41","doi-asserted-by":"publisher","unstructured":"Yang, X., Chen, Y., Eide, E., Regehr, J.: Finding and understanding bugs in Ccompilers. In: Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2011, San Jose, CA, USA, June 4-8, 2011. pp. 283\u2013294 (2011). https:\/\/doi.org\/10.1145\/1993498.1993532, http:\/\/www.cs.utah.edu\/~regehr\/papers\/pldi11-preprint.pdf","DOI":"10.1145\/1993498.1993532"},{"key":"10_CR42","unstructured":"Yun, I., Lee, S., Xu, M., Jang, Y., Kim, T.: QSYM : A practical concolic execution engine tailored for hybrid fuzzing. In: 27th USENIX Security Symposium (USENIX Security 18). pp. 745\u2013761. USENIX Association, Baltimore, MD (2018), https:\/\/www.usenix.org\/conference\/usenixsecurity18\/presentation\/yun"},{"key":"10_CR43","unstructured":"Zeller, A., Gopinath, R., B\u00f6hme, M., Fraser, G., Holler, C.: The fuzzing book. In: The Fuzzing Book. Saarland University (2019), https:\/\/www.fuzzingbook.org\/, retrieved 2019-09-09 16:42:54+02:00"}],"container-title":["Lecture Notes in Computer Science","Programming Languages and Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-030-72019-3_10","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,10,25]],"date-time":"2021-10-25T03:05:40Z","timestamp":1635131140000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-030-72019-3_10"}},"subtitle":["Combining Combinatorial and Property-Based Testing"],"short-title":[],"issued":{"date-parts":[[2021]]},"ISBN":["9783030720186","9783030720193"],"references-count":43,"URL":"https:\/\/doi.org\/10.1007\/978-3-030-72019-3_10","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"type":"print","value":"0302-9743"},{"type":"electronic","value":"1611-3349"}],"subject":[],"published":{"date-parts":[[2021]]},"assertion":[{"value":"23 March 2021","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"ESOP","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"European Symposium on Programming","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Luxembourg City","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Luxembourg","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2021","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"27 March 2021","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"1 April 2021","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"30","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"esop2021","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/etaps.org\/2021\/esop","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Single-blind","order":1,"name":"type","label":"Type","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"EasyChair","order":2,"name":"conference_management_system","label":"Conference Management System","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"79","order":3,"name":"number_of_submissions_sent_for_review","label":"Number of Submissions Sent for Review","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"24","order":4,"name":"number_of_full_papers_accepted","label":"Number of Full Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"0","order":5,"name":"number_of_short_papers_accepted","label":"Number of Short Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"30% - The value is computed by the equation \"Number of Full Papers Accepted \/ Number of Submissions Sent for Review * 100\" and then rounded to a whole number.","order":6,"name":"acceptance_rate_of_full_papers","label":"Acceptance Rate of Full Papers","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"3-5","order":7,"name":"average_number_of_reviews_per_paper","label":"Average Number of Reviews per Paper","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"10","order":8,"name":"average_number_of_papers_per_reviewer","label":"Average Number of Papers per Reviewer","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"Yes","order":9,"name":"external_reviewers_involved","label":"External Reviewers Involved","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"The conference took place virtually due to the COVID-19 pandemic","order":10,"name":"additional_info_on_review_process","label":"Additional Info on Review Process","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}}]}}