{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T01:23:22Z","timestamp":1760059402823,"version":"build-2065373602"},"reference-count":46,"publisher":"Association for Computing Machinery (ACM)","issue":"OOPSLA2","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Program. Lang."],"published-print":{"date-parts":[[2025,10,9]]},"abstract":"<jats:p>Property-based testing (PBT) is a testing methodology with origins in the functional programming community.  \nIn recent years, PBT libraries have been developed for non-functional languages, including Python. However,  \nto date, there is little evidence regarding how effective property-based tests are at finding bugs, and whether  \nsome kinds of property-based tests might be more effective than others. To gather this evidence, we conducted  \na corpus study of 426 Python programs that use Hypothesis, Python\u2019s most popular library for PBT. We  \ndeveloped formal definitions for 12 categories of property-based test and implemented an intraprocedural  \nstatic analysis that categorizes tests. Then, we evaluated the efficacy of test suites of 40 projects using mutation  \ntesting, and found that on average, each property-based test finds about 50 times as many mutations as the  \naverage unit test. We also identified the categories with the tests most effective at finding mutations, finding  \nthat tests that look for exceptions, that test inclusion in collections, and that check types are over 19 times  \nmore effective at finding mutations than other kinds of property-based tests. Finally, we conducted a parameter  \nsweep study to assess the strength of property-based tests as a function of the number of random inputs  \ngenerated, finding that 76% of mutations found were found within the first 20 inputs.<\/jats:p>","DOI":"10.1145\/3764068","type":"journal-article","created":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T08:49:50Z","timestamp":1759999790000},"page":"3897-3923","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["An Empirical Evaluation of Property-Based Testing in Python"],"prefix":"10.1145","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-9756-8832","authenticated-orcid":false,"given":"Savitha","family":"Ravi","sequence":"first","affiliation":[{"name":"University of California San Diego, La Jolla, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9369-4069","authenticated-orcid":false,"given":"Michael","family":"Coblenz","sequence":"additional","affiliation":[{"name":"University of California San Diego, La Jolla, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,10,9]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","unstructured":"2024. Code for mutation testing tool.. https:\/\/doi.org\/10.5281\/zenodo.16912461 10.5281\/zenodo.16912461","DOI":"10.5281\/zenodo.16912461"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","unstructured":"2024. Paper artifact.. https:\/\/doi.org\/10.5281\/zenodo.16921991 10.5281\/zenodo.16921991","DOI":"10.5281\/zenodo.16921991"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP52600.2021.00036"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/SEFM.2004.1347524"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2018.08.010"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.57873\/T34W2R"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416667"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/357766.351266"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME58846.2023.00049"},{"key":"e_1_2_2_10_1","unstructured":"Hypothesis Developers. [n. d.]. Hypothesis. https:\/\/hypothesis.works\/"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2013.6606588"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/263244.263267"},{"key":"e_1_2_2_13_1","unstructured":"The Python Language Foundation. 2024. The Python Language Reference: Python Full Grammar Specification version 3.13.0. https:\/\/docs.python.org\/3\/reference\/grammar.html"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2685612"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639581"},{"key":"e_1_2_2_16_1","volume-title":"Proc. Workshop on the Human Aspects of Types and Reasoning Assistants (HATRA).","author":"Goldstein Harrison","year":"2022","unstructured":"Harrison Goldstein, Joseph W Cutler, Adam Stein, Benjamin C Pierce, and Andrew Head. 2022. Some Problems with Properties. In Proc. Workshop on the Human Aspects of Types and Reasoning Assistants (HATRA)."},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3654777.3676407"},{"key":"e_1_2_2_18_1","unstructured":"Anders Hovm\u00f6ller and Many Contributors. 2024. mutmut. https:\/\/mutmut.readthedocs.io\/en\/latest\/"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-47147-7_4"},{"key":"e_1_2_2_20_1","unstructured":"SAS Institute Inc. 2024. JMP."},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568271"},{"volume-title":"Python Developers Survey 2023 Results","year":"2023","key":"e_1_2_2_22_1","unstructured":"JetBrains. 2023. Python Developers Survey 2023 Results, 2023.. https:\/\/lp.jetbrains.com\/python-developers-survey-2023\/."},{"key":"e_1_2_2_23_1","unstructured":"jqwik team. 2024. Jqwik. https:\/\/jqwik.net"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635929"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1093\/oxfordjournals.pan.a004868"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3243734.3243804"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/SBFT59156.2023.00015"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/QSIC.2014.33"},{"key":"e_1_2_2_29_1","unstructured":"Holger Krekel and pytest-dev team. 2015. pytest. https:\/\/docs.pytest.org\/en\/stable\/index.html"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3158133"},{"key":"e_1_2_2_31_1","volume-title":"Pierce","author":"Lampropoulos Leonidas","year":"2023","unstructured":"Leonidas Lampropoulos and Benjamin C. Pierce. 2023. QuickChick: Property-Based Testing in Coq. https:\/\/softwarefoundations.cis.upenn.edu\/qc-current\/index.html"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.21105\/joss.01891"},{"key":"e_1_2_2_33_1","unstructured":"David R. MacIver. 2025. Hypothesis Documentation. https:\/\/hypothesis.readthedocs.io\/en\/latest\/settings.html##hypothesis.settings.load_profile"},{"key":"e_1_2_2_34_1","unstructured":"Russell Mull. 2021. Effective Property-Based Testing. https:\/\/blog.auxon.io\/2021\/02\/01\/effective-property-based-testing\/"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/292540.292561"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3339002"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180183"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-22102-1_22"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-21251-2_2"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2015.86"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3607860"},{"key":"e_1_2_2_42_1","unstructured":"TIOBE. 2024. TIOBE Index. October 2024.. https:\/\/www.tiobe.com\/tiobe-index\/"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME58944.2024.00067"},{"key":"e_1_2_2_44_1","unstructured":"Scott Wlaschin. 2014. Choosing properties for property-based testing. https:\/\/fsharpforfunandprofit.com\/posts\/property-based-testing-2\/"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1016\/0164-1212(94)00098-0"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.25080\/majora-37cdd580-016"}],"container-title":["Proceedings of the ACM on Programming Languages"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3764068","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:51:40Z","timestamp":1760032300000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3764068"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,9]]},"references-count":46,"journal-issue":{"issue":"OOPSLA2","published-print":{"date-parts":[[2025,10,9]]}},"alternative-id":["10.1145\/3764068"],"URL":"https:\/\/doi.org\/10.1145\/3764068","relation":{},"ISSN":["2475-1421"],"issn-type":[{"type":"electronic","value":"2475-1421"}],"subject":[],"published":{"date-parts":[[2025,10,9]]},"assertion":[{"value":"2024-10-16","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-08-12","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-10-09","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}