{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,11]],"date-time":"2026-04-11T02:13:34Z","timestamp":1775873614685,"version":"3.50.1"},"reference-count":51,"publisher":"Association for Computing Machinery (ACM)","issue":"ICFP","license":[{"start":{"date-parts":[[2023,8,30]],"date-time":"2023-08-30T00:00:00Z","timestamp":1693353600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1955610, 2145649"],"award-info":[{"award-number":["1955610, 2145649"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Program. Lang."],"published-print":{"date-parts":[[2023,8,30]]},"abstract":"<jats:p>Property-based testing is a mainstay of functional programming, boasting a rich literature, an enthusiastic user community, and an abundance of tools \u2014 so many, indeed, that new users may have difficulty choosing. Moreover, any given framework may support a variety of strategies for generating test inputs; even experienced users may wonder which are better in a given situation. Sadly, the PBT literature, though long on creativity, is short on rigorous comparisons to help answer such questions.<\/jats:p>\n          <jats:p>We present Etna, a platform for empirical evaluation and comparison of PBT techniques. Etna incorporates a number of popular PBT frameworks and testing workloads from the literature, and its extensible architecture makes adding new ones easy, while handling the technical drudgery of performance measurement. To illustrate its benefits, we use Etna to carry out several experiments with popular PBT approaches in both Coq and Haskell, allowing users to more clearly understand best practices and tradeoffs.<\/jats:p>","DOI":"10.1145\/3607860","type":"journal-article","created":{"date-parts":[[2023,8,31]],"date-time":"2023-08-31T17:40:31Z","timestamp":1693503631000},"page":"878-894","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["Etna: An Evaluation Platform for Property-Based Testing (Experience Report)"],"prefix":"10.1145","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1507-1122","authenticated-orcid":false,"given":"Jessica","family":"Shi","sequence":"first","affiliation":[{"name":"University of Pennsylvania, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-5734-3598","authenticated-orcid":false,"given":"Alperen","family":"Keles","sequence":"additional","affiliation":[{"name":"University of Maryland, College Park, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9631-1169","authenticated-orcid":false,"given":"Harrison","family":"Goldstein","sequence":"additional","affiliation":[{"name":"University of Pennsylvania, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7839-1636","authenticated-orcid":false,"given":"Benjamin C.","family":"Pierce","sequence":"additional","affiliation":[{"name":"University of Pennsylvania, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0269-9815","authenticated-orcid":false,"given":"Leonidas","family":"Lampropoulos","sequence":"additional","affiliation":[{"name":"University of Maryland, College Park, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,8,31]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Alexandr Andoni Dumitru Daniliuc Sarfraz Khurshid and Darko Marinov. 2002. Evaluating the \"Small Scope Hypothesis\". Oct."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1411273.1411275"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-35308-6_10"},{"key":"e_1_2_1_5_1","first-page":"28716","volume-title":"Smart Testing of Functional Programs in Isabelle. In 18th International Conference on Logic for Programming, Artificial Intelligence, and Reasoning (Lecture Notes in Computer Science","volume":"167","author":"Bulwahn Lukas","year":"2012","unstructured":"Lukas Bulwahn. 2012. Smart Testing of Functional Programs in Isabelle. In 18th International Conference on Logic for Programming, Artificial Intelligence, and Reasoning (Lecture Notes in Computer Science, Vol. 7180). Springer, 153\u2013167. isbn:978-3-642-28716-9"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-07151-0_2"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/351240.351266"},{"key":"e_1_2_1_8_1","unstructured":"Simon Cruanes. 2017. QuickCheck Inspired Property-Based Testing for OCaml. https:\/\/github.com\/c-cube\/qcheck\/"},{"key":"e_1_2_1_9_1","unstructured":"Stephen Dolan. 2017. Property Fuzzing for OCaml. https:\/\/github.com\/stedolan\/crowbar"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2016.15"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2364506.2364515"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-662-46669-8_16"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-72019-3_10"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568278"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3428334"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.6778257"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2544174.2500574"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1017\/S0956796816000058"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-47147-7_4"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2010.62"},{"key":"e_1_2_1_21_1","unstructured":"Project Jupyter. 2023. Project Jupyter. https:\/\/jupyter.org"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3243734.3243804"},{"key":"e_1_2_1_23_1","volume-title":"Randomized Testing in PLT Redex. In Workshop on Scheme and Functional Programming.","author":"Klein Casey","year":"2009","unstructured":"Casey Klein and Robert Bruce Findler. 2009. Randomized Testing in PLT Redex. In Workshop on Scheme and Functional Programming."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2637647.2637655"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2814270.2814323"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3009837.3009868"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3360607"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3158133"},{"key":"e_1_2_1_30_1","volume-title":"Pierce","author":"Lampropoulos Leonidas","year":"2018","unstructured":"Leonidas Lampropoulos and Benjamin C. Pierce. 2018. QuickChick: Property-Based Testing In Coq. Electronic textbook."},{"key":"e_1_2_1_31_1","first-page":"196","volume-title":"Property Directed Generation of First-Order Test Data. In 8th Symposium on Trends in Functional Programming. Intellect, 105\u2013123","author":"Lindblad Fredrik","year":"2007","unstructured":"Fredrik Lindblad. 2007. Property Directed Generation of First-Order Test Data. In 8th Symposium on Trends in Functional Programming. Intellect, 105\u2013123. isbn:978-1-84150-196-3"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2018.00017"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3092703.3092711"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.4230\/LIPIcs.ECOOP.2020.13"},{"key":"e_1_2_1_35_1","volume-title":"Hypothesis: Property-Based Testing for Python. https:\/\/hypothesis.works\/","author":"MacIver David R.","year":"2016","unstructured":"David R. MacIver. 2016. Hypothesis: Property-Based Testing for Python. https:\/\/hypothesis.works\/"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3110259"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/AST.2019.00013"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3412932.3412943"},{"key":"e_1_2_1_39_1","unstructured":"Rickard Nilsson. 2019. ScalaCheck: Property-Based Testing for Scala. https:\/\/scalacheck.org\/"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330576"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1982595.1982615"},{"key":"e_1_2_1_42_1","unstructured":"pandas. 2023. pandas - Python Data Analysis Library. https:\/\/pandas.pydata.org\/"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2034654.2034663"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3519939.3523707"},{"key":"e_1_2_1_45_1","unstructured":"Benjamin C. Pierce. 2018. Software Foundations. Electronic textbook."},{"key":"e_1_2_1_46_1","volume-title":"Plotly: Low-Code Data App Development. https:\/\/plotly.com\/","year":"2023","unstructured":"Plotly. 2023. Plotly: Low-Code Data App Development. https:\/\/plotly.com\/"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/1543134.1411292"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2599030"},{"key":"e_1_2_1_49_1","volume-title":"Hedgehog: Release with Confidence. https:\/\/hackage.haskell.org\/package\/hedgehog\/","author":"Stanley Jacob","year":"2019","unstructured":"Jacob Stanley. 2019. Hedgehog: Release with Confidence. https:\/\/hackage.haskell.org\/package\/hedgehog\/"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025768"},{"key":"e_1_2_1_51_1","unstructured":"Li-yao Xia. 2018. A quick tour of generic-random. https:\/\/hackage.haskell.org\/package\/generic-random-1.5.0.0\/docs\/Generic-Random.html"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/1993498.1993532"},{"key":"e_1_2_1_53_1","first-page":"31","volume-title":"FIXREVERTER: A Realistic Bug Injection Methodology for Benchmarking Fuzz Testing. In 31st USENIX Security Symposium. USENIX Association, 3699\u20133715","author":"Zhang Zenong","year":"2022","unstructured":"Zenong Zhang, Zach Patterson, Michael Hicks, and Shiyi Wei. 2022. FIXREVERTER: A Realistic Bug Injection Methodology for Benchmarking Fuzz Testing. In 31st USENIX Security Symposium. USENIX Association, 3699\u20133715. isbn:978-1-939133-31-1"}],"container-title":["Proceedings of the ACM on Programming Languages"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3607860","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3607860","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3607860","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:37:06Z","timestamp":1750178226000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3607860"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,30]]},"references-count":51,"journal-issue":{"issue":"ICFP","published-print":{"date-parts":[[2023,8,30]]}},"alternative-id":["10.1145\/3607860"],"URL":"https:\/\/doi.org\/10.1145\/3607860","relation":{},"ISSN":["2475-1421"],"issn-type":[{"value":"2475-1421","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,30]]},"assertion":[{"value":"2023-08-31","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}