{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T11:34:43Z","timestamp":1791459283127,"version":"4.3.4"},"reference-count":117,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2020,9,26]],"date-time":"2020-09-26T00:00:00Z","timestamp":1601078400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2020,10,31]]},"abstract":"<jats:p>\n                    Raw lines of code (LOC) is a metric that does not, at first glance, seem extremely useful for automated test generation. It is both highly language-dependent and not extremely meaningful, semantically, within a language: one coder can produce the same effect with many fewer lines than another. However,\n                    <jats:italic>relative LOC<\/jats:italic>\n                    , between components of the same project, turns out to be a highly useful metric for automated testing. In this article, we make use of a heuristic based on LOC counts for tested functions to dramatically improve the effectiveness of automated test generation. This approach is particularly valuable in languages where collecting code coverage data to guide testing has a very high overhead. We apply the heuristic to property-based Python testing using the TSTL (Template Scripting Testing Language) tool. In our experiments, the simple LOC heuristic can improve branch and statement coverage by large margins (often more than 20%, up to 40% or more) and improve fault detection by an even larger margin (usually more than 75% and up to 400% or more). The LOC heuristic is also easy to combine with other approaches and is comparable to, and possibly more effective than, two well-established approaches for guiding random testing.\n                  <\/jats:p>","DOI":"10.1145\/3408896","type":"journal-article","created":{"date-parts":[[2020,9,26]],"date-time":"2020-09-26T06:58:57Z","timestamp":1601103537000},"page":"1-38","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":12,"title":["Using Relative Lines of Code to Guide Automated Test Generation for Python"],"prefix":"10.1145","volume":"29","author":[{"given":"Josie","family":"Holmes","sequence":"first","affiliation":[{"name":"School of Informatics, Computing 8 Cyber Systems, Northern Arizona University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8221-5352","authenticated-orcid":false,"given":"Iftekhar","family":"Ahmed","sequence":"additional","affiliation":[{"name":"Donald Bren School of Information and Computer Sciences, University of California, Irvine"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Caius","family":"Brindescu","sequence":"additional","affiliation":[{"name":"School of Electrical Engineering and Computer, Oregon State University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rahul","family":"Gopinath","sequence":"additional","affiliation":[{"name":"Center for IT-Security, Privacy and Accountability (CISPA), University of Saarbr\u00fccken"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"He","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Electrical Engineering and Computer, Oregon State University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alex","family":"Groce","sequence":"additional","affiliation":[{"name":"School of Informatics, Computing 8 Cyber Systems, Northern Arizona University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,9,26]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-47106-8_22"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/174675.175935"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2950290.2950324"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2931037.2931056"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2007.1005"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1321631.1321654"},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the International Conference on Software Engineering. 402--411","author":"Andrews James H.","unstructured":"James H. Andrews , L. C. Briand , and Y. Labiche . 2005. Is mutation an appropriate tool for testing experiments? In Proceedings of the International Conference on Software Engineering. 402--411 . James H. Andrews, L. C. Briand, and Y. Labiche. 2005. Is mutation an appropriate tool for testing experiments? In Proceedings of the International Conference on Software Engineering. 402--411."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1486"},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of the International Symposium on Software Testing and Analysis. 219--230","author":"Arcuri Andrea","unstructured":"Andrea Arcuri , Muhammad Zohaib Z. Iqbal , and Lionel C. Briand . 2010. Formal analysis of the effectiveness and predictability of random testing . In Proceedings of the International Symposium on Software Testing and Analysis. 219--230 . Andrea Arcuri, Muhammad Zohaib Z. Iqbal, and Lionel C. Briand. 2010. Formal analysis of the effectiveness and predictability of random testing. In Proceedings of the International Symposium on Software Testing and Analysis. 219--230."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.979986"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.544352"},{"key":"e_1_2_1_13_1","unstructured":"Ned Batchelder. 2015. Coverage.py. Retrieved from https:\/\/coverage.readthedocs.org\/en\/coverage-4.0.1\/.  Ned Batchelder. 2015. Coverage.py. Retrieved from https:\/\/coverage.readthedocs.org\/en\/coverage-4.0.1\/."},{"key":"e_1_2_1_14_1","volume-title":"Bieman and Byung-Kyoo Kang","author":"James","year":"1995","unstructured":"James M. Bieman and Byung-Kyoo Kang . 1995 . Cohesion and reuse in an object-oriented system. SIGSOFT Softw. Eng. Notes 20, SI (Aug . 1995), 259--262. DOI:https:\/\/doi.org\/10.1145\/223427.211856 10.1145\/223427.211856 James M. Bieman and Byung-Kyoo Kang. 1995. Cohesion and reuse in an object-oriented system. SIGSOFT Softw. Eng. Notes 20, SI (Aug. 1995), 259--262. DOI:https:\/\/doi.org\/10.1145\/223427.211856"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635923"},{"key":"e_1_2_1_16_1","volume-title":"Briand and J\u00fcrgen W\u00fcst","author":"Lionel","year":"2002","unstructured":"Lionel C. Briand and J\u00fcrgen W\u00fcst . 2002 . Empirical studies of quality models in object-oriented systems. Advances in Computers, Vol. 56 . Elsevier , 97--166. DOI:https:\/\/doi.org\/10.1016\/S0065-2458(02)80005-5 10.1016\/S0065-2458(02)80005-5 Lionel C. Briand and J\u00fcrgen W\u00fcst. 2002. Empirical studies of quality models in object-oriented systems. Advances in Computers, Vol. 56. Elsevier, 97--166. DOI:https:\/\/doi.org\/10.1016\/S0065-2458(02)80005-5"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/302405.302654"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1009815306478"},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the IEEE 6th International Conference on Software Testing, Verification and Validation Workshops. 74--79","author":"Buchgeher G.","year":"2013","unstructured":"G. Buchgeher , C. Ernstbrunner , R. Ramler , and M. Lusser . 2013. Towards tool-support for test case selection in manual regression testing . In Proceedings of the IEEE 6th International Conference on Software Testing, Verification and Validation Workshops. 74--79 . DOI:https:\/\/doi.org\/10.1109\/ICSTW. 2013 .16 10.1109\/ICSTW.2013.16 G. Buchgeher, C. Ernstbrunner, R. Ramler, and M. Lusser. 2013. Towards tool-support for test case selection in manual regression testing. In Proceedings of the IEEE 6th International Conference on Software Testing, Verification and Validation Workshops. 74--79. DOI:https:\/\/doi.org\/10.1109\/ICSTW.2013.16"},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the Conference on Operating System Design and Implementation. 209--224","author":"Cadar Cristian","year":"2008","unstructured":"Cristian Cadar , Daniel Dunbar , and Dawson Engler . 2008 . KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs . In Proceedings of the Conference on Operating System Design and Implementation. 209--224 . Cristian Cadar, Daniel Dunbar, and Dawson Engler. 2008. KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs. In Proceedings of the Conference on Operating System Design and Implementation. 209--224."},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the 27th IEEE International Conference on Software Maintenance (ICSM\u201911)","author":"Carlson R.","year":"2011","unstructured":"R. Carlson , H. Do , and A. Denton . 2011. A clustering approach to improving test case prioritization: An industrial case study . In Proceedings of the 27th IEEE International Conference on Software Maintenance (ICSM\u201911) . 382--391. DOI:https:\/\/doi.org\/10.1109\/ICSM. 2011 .6080805 10.1109\/ICSM.2011.6080805 R. Carlson, H. Do, and A. Denton. 2011. A clustering approach to improving test case prioritization: An industrial case study. In Proceedings of the 27th IEEE International Conference on Software Maintenance (ICSM\u201911). 382--391. DOI:https:\/\/doi.org\/10.1109\/ICSM.2011.6080805"},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the 28th USENIX Security Symposium (USENIX Security\u201919)","author":"Chen Yuanliang","year":"2019","unstructured":"Yuanliang Chen , Yu Jiang , Fuchen Ma , Jie Liang , Mingzhe Wang , Chijin Zhou , Xun Jiao , and Zhuo Su . 2019 . EnFuzz: Ensemble fuzzing with seed synchronization among diverse fuzzers . In Proceedings of the 28th USENIX Security Symposium (USENIX Security\u201919) . 1967--1983. Yuanliang Chen, Yu Jiang, Fuchen Ma, Jie Liang, Mingzhe Wang, Chijin Zhou, Xun Jiao, and Zhuo Su. 2019. EnFuzz: Ensemble fuzzing with seed synchronization among diverse fuzzers. In Proceedings of the 28th USENIX Security Symposium (USENIX Security\u201919). 1967--1983."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2004.32"},{"key":"e_1_2_1_24_1","unstructured":"Travis CI. 2012. Customizing the Build: Build Timeouts. Retrieved from https:\/\/docs.travis-ci.com\/user\/customizing-the-build\/#Build-Timeouts.  Travis CI. 2012. Customizing the Build: Build Timeouts. Retrieved from https:\/\/docs.travis-ci.com\/user\/customizing-the-build\/#Build-Timeouts."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/351240.351266"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.879821"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.21236\/ADA459081"},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the International Conference on Software Quality (QSIC\u201994)","author":"Abreu Fernando Brito","year":"1994","unstructured":"Fernando Brito e Abreu and Rog\u00e9rio Carapu\u00e7a . 1994 . Object-oriented software engineering: Measuring and controlling the development process . In Proceedings of the International Conference on Software Quality (QSIC\u201994) . Fernando Brito e Abreu and Rog\u00e9rio Carapu\u00e7a. 1994. Object-oriented software engineering: Measuring and controlling the development process. In Proceedings of the International Conference on Software Quality (QSIC\u201994)."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/METRIC.1996.492446"},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME\u201915)","author":"Ekelund E. D.","year":"2015","unstructured":"E. D. Ekelund and E. Engstr\u00f6m . 2015. Efficient regression testing based on test history: An industrial evaluation . In Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME\u201915) . 449--457. DOI:https:\/\/doi.org\/10.1109\/ICSM. 2015 .7332496 10.1109\/ICSM.2015.7332496 E. D. Ekelund and E. Engstr\u00f6m. 2015. Efficient regression testing based on test history: An industrial evaluation. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME\u201915). 449--457. DOI:https:\/\/doi.org\/10.1109\/ICSM.2015.7332496"},{"key":"e_1_2_1_31_1","first-page":"7","article-title":"The confounding effect of class size on the validity of object-oriented metrics","volume":"27","author":"Emam Kalhed El","year":"2001","unstructured":"Kalhed El Emam , Sa\u00efda Benlarbi , Nishith Goel , and Shesh N. Rai . 2001 . The confounding effect of class size on the validity of object-oriented metrics . IEEE Trans. Softw. Eng. 27 , 7 (July 2001), 630--650. DOI:https:\/\/doi.org\/10.1109\/32.935855 10.1109\/32.935855 Kalhed El Emam, Sa\u00efda Benlarbi, Nishith Goel, and Shesh N. Rai. 2001. The confounding effect of class size on the validity of object-oriented metrics. IEEE Trans. Softw. Eng. 27, 7 (July 2001), 630--650. DOI:https:\/\/doi.org\/10.1109\/32.935855","journal-title":"IEEE Trans. Softw. Eng."},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of the 3rd International Conference on Software Testing, Verification and Validation. 75--78","author":"Engstr\u00f6m E.","year":"2010","unstructured":"E. Engstr\u00f6m , P. Runeson , and G. Wikstrand . 2010. An empirical evaluation of regression testing based on fix-cache recommendations . In Proceedings of the 3rd International Conference on Software Testing, Verification and Validation. 75--78 . DOI:https:\/\/doi.org\/10.1109\/ICST. 2010 .40 10.1109\/ICST.2010.40 E. Engstr\u00f6m, P. Runeson, and G. Wikstrand. 2010. An empirical evaluation of regression testing based on fix-cache recommendations. In Proceedings of the 3rd International Conference on Software Testing, Verification and Validation. 75--78. DOI:https:\/\/doi.org\/10.1109\/ICST.2010.40"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/SCAM.2004.11"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.879815"},{"key":"e_1_2_1_35_1","unstructured":"M. Fowler. 2010. Domain-specific Languages. Addison-Wesley Professional.  M. Fowler. 2010. Domain-specific Languages. Addison-Wesley Professional."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2025113.2025179"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3194718.3194719"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2483760.2483769"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642937.2643019"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/1065010.1065036"},{"key":"e_1_2_1_41_1","unstructured":"Peter Goodman. 2016. A fuzzer and a symbolic executor walk into a cloud. Retrieved from https:\/\/blog.trailofbits.com\/2016\/08\/02\/engineering-solutions-to-hard-program-analysis-problems\/.  Peter Goodman. 2016. A fuzzer and a symbolic executor walk into a cloud. Retrieved from https:\/\/blog.trailofbits.com\/2016\/08\/02\/engineering-solutions-to-hard-program-analysis-problems\/."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568278"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2014.40"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.859533"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/2338966.2336800"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-34026-0_42"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2012.1"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2007.68"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-17524-9_15"},{"key":"e_1_2_1_50_1","volume-title":"TSTL: The template scripting testing language.","author":"Groce Alex","year":"2015","unstructured":"Alex Groce , Jervis Pinto , Pooria Azimi , Pranjal Mittal , Josie Holmes , and Kevin Kellar . 2015 . TSTL: The template scripting testing language. Retrieved from https:\/\/github.com\/agroce\/tstl. Alex Groce, Jervis Pinto, Pooria Azimi, Pranjal Mittal, Josie Holmes, and Kevin Kellar. 2015. TSTL: The template scripting testing language. Retrieved from https:\/\/github.com\/agroce\/tstl."},{"key":"e_1_2_1_51_1","volume-title":"Heuristics for model checking Java programs. Softw. Tools Technol. Transf. 6(4)","author":"Groce Alex","year":"2004","unstructured":"Alex Groce and Willem Visser . 2004. Heuristics for model checking Java programs. Softw. Tools Technol. Transf. 6(4) ( 2004 ), 260--276. Alex Groce and Willem Visser. 2004. Heuristics for model checking Java programs. Softw. Tools Technol. Transf. 6(4) (2004), 260--276."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/2338965.2336763"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2005.112"},{"key":"e_1_2_1_54_1","volume-title":"Encyclopedia of Software Engineering","author":"Hamlet Richard","unstructured":"Richard Hamlet . 1994. Random testing . In Encyclopedia of Software Engineering . Wiley , 970--978. Richard Hamlet. 1994. Random testing. In Encyclopedia of Software Engineering. Wiley, 970--978."},{"key":"e_1_2_1_55_1","volume-title":"Proceedings of the IEEE International Working Conference on Source Code Analysis and Manipulation.","author":"Harman Mark","year":"2018","unstructured":"Mark Harman and Peter O\u2019Hearn . 2018 . From start-ups to scale-ups: Open problems, challenges and myths in static and dynamic program analysis for testing and verification . In Proceedings of the IEEE International Working Conference on Source Code Analysis and Manipulation. Mark Harman and Peter O\u2019Hearn. 2018. From start-ups to scale-ups: Open problems, challenges and myths in static and dynamic program analysis for testing and verification. In Proceedings of the IEEE International Working Conference on Source Code Analysis and Manipulation."},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.66"},{"key":"e_1_2_1_57_1","volume-title":"Proceedings of the Gemeinsamer Tagungsband der Workshops der Tagung Software Engineering (SE\u201916)","author":"Hirzel Matthias","year":"2016","unstructured":"Matthias Hirzel and Herbert Klaeren . 2016 . Graph-walk-based selective regression testing of web applications created with Google web toolkit . In Proceedings of the Gemeinsamer Tagungsband der Workshops der Tagung Software Engineering (SE\u201916) . 55--69. Retrieved from: http:\/\/ceur-ws.org\/Vol-1559\/paper05.pdf. Matthias Hirzel and Herbert Klaeren. 2016. Graph-walk-based selective regression testing of web applications created with Google web toolkit. In Proceedings of the Gemeinsamer Tagungsband der Workshops der Tagung Software Engineering (SE\u201916). 55--69. Retrieved from: http:\/\/ceur-ws.org\/Vol-1559\/paper05.pdf."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10009-016-0445-y"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2008.9"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-85114-1_11"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2010.110"},{"key":"e_1_2_1_62_1","unstructured":"Laura Inozemtseva. [n.d.]. Supplemental results for \u201cCoverage is not Correlated...\u201d. DOI:http:\/\/inozemtseva.com\/research\/2014\/icse\/coverage  Laura Inozemtseva. [n.d.]. Supplemental results for \u201cCoverage is not Correlated...\u201d. DOI:http:\/\/inozemtseva.com\/research\/2014\/icse\/coverage"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568271"},{"key":"e_1_2_1_64_1","volume-title":"Proceedings of the International Symposium on Software Testing and Analysis. ACM, 437--440","author":"Just Ren\u00e9","unstructured":"Ren\u00e9 Just , Darioush Jalali , and Michael D. Ernst . 2014. Defects4J: A database of existing faults to enable controlled testing studies for Java programs . In Proceedings of the International Symposium on Software Testing and Analysis. ACM, 437--440 . Ren\u00e9 Just, Darioush Jalali, and Michael D. Ernst. 2014. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the International Symposium on Software Testing and Analysis. ACM, 437--440."},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3243734.3243804"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2008.90"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1002\/smr.1760"},{"key":"e_1_2_1_69_1","volume-title":"Hypothesis: Test faster, fix more.","author":"MacIver David R.","year":"2013","unstructured":"David R. MacIver . 2013 . Hypothesis: Test faster, fix more. Retrieved from http:\/\/hypothesis.works\/. David R. MacIver. 2013. Hypothesis: Test faster, fix more. Retrieved from http:\/\/hypothesis.works\/."},{"key":"e_1_2_1_70_1","unstructured":"David R. MacIver. 2016. Rule Based Stateful Testing. Retrieved from http:\/\/hypothesis.works\/articles\/rule-based-stateful-testing\/.  David R. MacIver. 2016. Rule Based Stateful Testing. Retrieved from http:\/\/hypothesis.works\/articles\/rule-based-stateful-testing\/."},{"key":"e_1_2_1_71_1","unstructured":"David R. MacIver. 2017. Python Coverage could be fast. Retrieved from https:\/\/www.drmaciver.com\/2017\/09\/python-coverage-could-be-fast\/.  David R. MacIver. 2017. Python Coverage could be fast. Retrieved from https:\/\/www.drmaciver.com\/2017\/09\/python-coverage-could-be-fast\/."},{"key":"e_1_2_1_72_1","unstructured":"David R. MacIver. 2017. Coverage adds a lot of overhead when the base test is fast. Retrieved from https:\/\/github.com\/HypothesisWorks\/hypothesis\/issues\/914.  David R. MacIver. 2017. Coverage adds a lot of overhead when the base test is fast. Retrieved from https:\/\/github.com\/HypothesisWorks\/hypothesis\/issues\/914."},{"key":"e_1_2_1_73_1","unstructured":"David R. MacIver and PyPI. 2015. Usage stats for hypothesis on PyPI. Retrieved from https:\/\/libraries.io\/pypi\/hypothesis\/usage.  David R. MacIver and PyPI. 2015. Usage stats for hypothesis on PyPI. Retrieved from https:\/\/libraries.io\/pypi\/hypothesis\/usage."},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2007.70768"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1109\/QRS.2015.31"},{"key":"e_1_2_1_76_1","volume-title":"Proceedings of the IEEE International Conference on Software Maintenance. 540--543","author":"Marijan D.","year":"2013","unstructured":"D. Marijan , A. Gotlieb , and S. Sen . 2013. Test case prioritization for continuous regression testing: An industrial case study . In Proceedings of the IEEE International Conference on Software Maintenance. 540--543 . DOI:https:\/\/doi.org\/10.1109\/ICSM. 2013 .91 10.1109\/ICSM.2013.91 D. Marijan, A. Gotlieb, and S. Sen. 2013. Test case prioritization for continuous regression testing: An industrial case study. In Proceedings of the IEEE International Conference on Software Maintenance. 540--543. DOI:https:\/\/doi.org\/10.1109\/ICSM.2013.91"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2012.6227146"},{"key":"e_1_2_1_78_1","first-page":"4","article-title":"A complexity measure","volume":"2","author":"McCabe T. J.","year":"1976","unstructured":"T. J. McCabe . 1976 . A complexity measure . IEEE Trans. Softw. Eng. 2 , 4 (July 1976), 308--320. DOI:https:\/\/doi.org\/10.1109\/TSE.1976.233837 10.1109\/TSE.1976.233837 T. J. McCabe. 1976. A complexity measure. IEEE Trans. Softw. Eng. 2, 4 (July 1976), 308--320. DOI:https:\/\/doi.org\/10.1109\/TSE.1976.233837","journal-title":"IEEE Trans. Softw. Eng."},{"key":"e_1_2_1_79_1","volume-title":"Differential testing for software. Dig. Tech. J. Dig. Equip","author":"McKeeman William","year":"1998","unstructured":"William McKeeman . 1998. Differential testing for software. Dig. Tech. J. Dig. Equip . Corp . 10(1) ( 1998 ), 100--107. William McKeeman. 1998. Differential testing for software. Dig. Tech. J. Dig. Equip. Corp. 10(1) (1998), 100--107."},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.5555\/1077276.1077279"},{"key":"e_1_2_1_81_1","volume-title":"Proceedings of the 27th NASA Goddard\/IEEE Software Engineering Workshop.51--57","author":"Menzies T.","year":"2002","unstructured":"T. Menzies , J. S. Di Stefano , M. Chapman , and K. McGill . 2002. Metrics that matter . In Proceedings of the 27th NASA Goddard\/IEEE Software Engineering Workshop.51--57 . DOI:https:\/\/doi.org\/10.1109\/SEW. 2002 .1199449 10.1109\/SEW.2002.1199449 T. Menzies, J. S. Di Stefano, M. Chapman, and K. McGill. 2002. Metrics that matter. In Proceedings of the 27th NASA Goddard\/IEEE Software Engineering Workshop.51--57. DOI:https:\/\/doi.org\/10.1109\/SEW.2002.1199449"},{"key":"e_1_2_1_82_1","unstructured":"Rickard Nilsson Shane Auckland Mark Sumner and Sanjiv Sahayam. 2016. ScalaCheck User Guide. Retrieved from https:\/\/github.com\/rickynils\/scalacheck\/blob\/master\/doc\/UserGuide.md.  Rickard Nilsson Shane Auckland Mark Sumner and Sanjiv Sahayam. 2016. ScalaCheck User Guide. Retrieved from https:\/\/github.com\/rickynils\/scalacheck\/blob\/master\/doc\/UserGuide.md."},{"key":"e_1_2_1_83_1","volume-title":"Untch","author":"Jefferson Offutt A.","year":"2001","unstructured":"A. Jefferson Offutt and Roland H . Untch . 2001 . Mutation 2000: Uniting the orthogonal. In Mutation Testing for the New Century. Springer , 34--44. A. Jefferson Offutt and Roland H. Untch. 2001. Mutation 2000: Uniting the orthogonal. In Mutation Testing for the New Century. Springer, 34--44."},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1145\/2970276.2970351"},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2007.1015"},{"key":"e_1_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2005.49"},{"key":"e_1_2_1_87_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2007.37"},{"key":"e_1_2_1_88_1","first-page":"10","article-title":"Empirical analysis of software fault content and fault proneness using Bayesian methods","volume":"33","author":"Pai G. J.","year":"2007","unstructured":"G. J. Pai and J. Bechta Dugan . 2007 . Empirical analysis of software fault content and fault proneness using Bayesian methods . IEEE Trans. Softw. Eng. 33 , 10 (Oct. 2007), 675--686. DOI:https:\/\/doi.org\/10.1109\/TSE.2007.70722 10.1109\/TSE.2007.70722 G. J. Pai and J. Bechta Dugan. 2007. Empirical analysis of software fault content and fault proneness using Bayesian methods. IEEE Trans. Softw. Eng. 33, 10 (Oct. 2007), 675--686. DOI:https:\/\/doi.org\/10.1109\/TSE.2007.70722","journal-title":"IEEE Trans. Softw. Eng."},{"key":"e_1_2_1_89_1","doi-asserted-by":"publisher","DOI":"10.1145\/2034654.2034663"},{"key":"e_1_2_1_90_1","doi-asserted-by":"publisher","DOI":"10.1145\/1993498.1993558"},{"key":"e_1_2_1_91_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2013.02.009"},{"key":"e_1_2_1_92_1","doi-asserted-by":"publisher","DOI":"10.14722\/ndss.2017.23404"},{"key":"e_1_2_1_93_1","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884848"},{"key":"e_1_2_1_94_1","volume-title":"Proceedings of the IEEE\/ACM 37th IEEE International Conference on Software Engineering","volume":"1","author":"Saha Ripon K.","unstructured":"Ripon K. Saha , Lingming Zhang , Sarfraz Khurshid , and Dewayne E. Perry . 2015. An information retrieval approach for regression test prioritization based on program changes . In Proceedings of the IEEE\/ACM 37th IEEE International Conference on Software Engineering , Vol. 1 . IEEE, 268--279. Ripon K. Saha, Lingming Zhang, Sarfraz Khurshid, and Dewayne E. Perry. 2015. An information retrieval approach for regression test prioritization based on program changes. In Proceedings of the IEEE\/ACM 37th IEEE International Conference on Software Engineering, Vol. 1. IEEE, 268--279."},{"key":"e_1_2_1_95_1","unstructured":"Scientific Toolworks Inc.2017. Understand\u2122 Static Code Analysis Tool. Retrieved from https:\/\/scitools.com\/.  Scientific Toolworks Inc.2017. Understand\u2122 Static Code Analysis Tool. Retrieved from https:\/\/scitools.com\/."},{"key":"e_1_2_1_96_1","volume-title":"Tutorial: How to collect test coverages for Rust project.","author":"Seonghoon Kang","year":"2015","unstructured":"Kang Seonghoon . 2015 . Tutorial: How to collect test coverages for Rust project. Retrieved from https:\/\/users.rust-lang.org\/t\/tutorial-how-to-collect-test-coverages-for-rust-project\/650. Kang Seonghoon. 2015. Tutorial: How to collect test coverages for Rust project. Retrieved from https:\/\/users.rust-lang.org\/t\/tutorial-how-to-collect-test-coverages-for-rust-project\/650."},{"key":"e_1_2_1_97_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2015.86"},{"key":"e_1_2_1_98_1","doi-asserted-by":"publisher","DOI":"10.1145\/2739480.2754696"},{"key":"e_1_2_1_99_1","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1660"},{"key":"e_1_2_1_100_1","volume-title":"Proceedings of the International Symposium on Empirical Software Engineering.. DOI:https:\/\/doi.org\/10","author":"Skoglund M.","year":"2005","unstructured":"M. Skoglund and P. Runeson . 2005. A case study of the class firewall regression test selection technique on a large scale distributed software system . In Proceedings of the International Symposium on Empirical Software Engineering.. DOI:https:\/\/doi.org\/10 .1109\/ISESE. 2005 .1541816 10.1109\/ISESE.2005.1541816 M. Skoglund and P. Runeson. 2005. A case study of the class firewall regression test selection technique on a large scale distributed software system. In Proceedings of the International Symposium on Empirical Software Engineering.. DOI:https:\/\/doi.org\/10.1109\/ISESE.2005.1541816"},{"key":"e_1_2_1_101_1","doi-asserted-by":"publisher","DOI":"10.1145\/566171.566187"},{"key":"e_1_2_1_102_1","volume-title":"Proceedings of the 33rd International Conference on Software Engineering (ICSE\u201911)","author":"Staats Matt","year":"1985","unstructured":"Matt Staats , Michael W. Whalen , and Mats P. E. Heimdahl . 2011. Programs, tests, and oracles: The foundations of testing revisited . In Proceedings of the 33rd International Conference on Software Engineering (ICSE\u201911) . 391--400. DOI:https:\/\/doi.org\/10.1145\/ 1985 793.1985847 10.1145\/1985793.1985847 Matt Staats, Michael W. Whalen, and Mats P. E. Heimdahl. 2011. Programs, tests, and oracles: The foundations of testing revisited. In Proceedings of the 33rd International Conference on Software Engineering (ICSE\u201911). 391--400. DOI:https:\/\/doi.org\/10.1145\/1985793.1985847"},{"key":"e_1_2_1_103_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2014.2361131"},{"key":"e_1_2_1_104_1","volume-title":"Information Technology: New Generations","author":"Tahvili Sahar","unstructured":"Sahar Tahvili , Wasif Afzal , Mehrdad Saadatmand , Markus Bohlin , Daniel Sundmark , and Stig Larsson . 2016. Towards earlier fault detection by value-driven prioritization of test cases using fuzzy TOPSIS . In Information Technology: New Generations . Springer , 745--759. Sahar Tahvili, Wasif Afzal, Mehrdad Saadatmand, Markus Bohlin, Daniel Sundmark, and Stig Larsson. 2016. Towards earlier fault detection by value-driven prioritization of test cases using fuzzy TOPSIS. In Information Technology: New Generations. Springer, 745--759."},{"key":"e_1_2_1_105_1","volume-title":"Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA\u201902)","author":"Mustafa","unstructured":"Mustafa M. Tikir and Jeffrey K. Hollingsworth. 2002. Efficient instrumentation for code coverage testing . In Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA\u201902) . ACM, New York, NY, 86--96. DOI:https:\/\/doi.org\/10.1145\/566172.566186 10.1145\/566172.566186 Mustafa M. Tikir and Jeffrey K. Hollingsworth. 2002. Efficient instrumentation for code coverage testing. In Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA\u201902). ACM, New York, NY, 86--96. DOI:https:\/\/doi.org\/10.1145\/566172.566186"},{"key":"e_1_2_1_106_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00048"},{"key":"e_1_2_1_107_1","unstructured":"user1689822. 2012. python AVL tree insertion. Retrieved from http:\/\/stackoverflow.com\/questions\/12537986\/python-avl-tree-insertion.  user1689822. 2012. python AVL tree insertion. Retrieved from http:\/\/stackoverflow.com\/questions\/12537986\/python-avl-tree-insertion."},{"key":"e_1_2_1_108_1","doi-asserted-by":"publisher","DOI":"10.1002\/smr.371"},{"key":"e_1_2_1_109_1","volume-title":"Proceedings of the 20th IEEE International Conference on Software Maintenance.18--27","author":"White L.","year":"2004","unstructured":"L. White and B. Robinson . 2004. Industrial real-time regression testing and analysis using firewalls . In Proceedings of the 20th IEEE International Conference on Software Maintenance.18--27 . DOI:https:\/\/doi.org\/10.1109\/ICSM. 2004 .1357786 10.1109\/ICSM.2004.1357786 L. White and B. Robinson. 2004. Industrial real-time regression testing and analysis using firewalls. In Proceedings of the 20th IEEE International Conference on Software Maintenance.18--27. DOI:https:\/\/doi.org\/10.1109\/ICSM.2004.1357786"},{"key":"e_1_2_1_110_1","volume-title":"Proceedings of the International Conference on Software Testing Verification and Validation. 299--302","author":"Wikstrand G.","year":"2009","unstructured":"G. Wikstrand , R. Feldt , J. K. Gorantla , W. Zhe , and C. White . 2009. Dynamic regression test selection based on a file cache an industrial evaluation . In Proceedings of the International Conference on Software Testing Verification and Validation. 299--302 . DOI:https:\/\/doi.org\/10.1109\/ICST. 2009 .42 10.1109\/ICST.2009.42 G. Wikstrand, R. Feldt, J. K. Gorantla, W. Zhe, and C. White. 2009. Dynamic regression test selection based on a file cache an industrial evaluation. In Proceedings of the International Conference on Software Testing Verification and Validation. 299--302. DOI:https:\/\/doi.org\/10.1109\/ICST.2009.42"},{"key":"e_1_2_1_111_1","doi-asserted-by":"publisher","DOI":"10.1093\/comjnl\/bxm021"},{"key":"e_1_2_1_112_1","doi-asserted-by":"publisher","DOI":"10.1145\/2771783.2771805"},{"key":"e_1_2_1_113_1","unstructured":"Michal Zalewski. 2014. american fuzzy lop (2.35b). Retrieved from http:\/\/lcamtuf.coredump.cx\/afl\/.  Michal Zalewski. 2014. american fuzzy lop (2.35b). Retrieved from http:\/\/lcamtuf.coredump.cx\/afl\/."},{"key":"e_1_2_1_114_1","doi-asserted-by":"publisher","DOI":"10.1145\/2610384.2610392"},{"key":"e_1_2_1_115_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2009.5306304"},{"key":"e_1_2_1_116_1","doi-asserted-by":"publisher","DOI":"10.1145\/1134285.1134357"},{"key":"e_1_2_1_117_1","doi-asserted-by":"publisher","DOI":"10.1145\/1321631.1321665"},{"key":"e_1_2_1_118_1","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1109\/TSE.2006.102","article-title":"Empirical analysis of object-oriented design metrics for predicting high and low severity faults","volume":"32","author":"Zhou Yuming","year":"2006","unstructured":"Yuming Zhou and Hareton Leung . 2006 . Empirical analysis of object-oriented design metrics for predicting high and low severity faults . IEEE Trans. Softw. Eng. 32 , 10 (Oct. 2006), 771--789. DOI:https:\/\/doi.org\/10.1109\/TSE.2006.102 10.1109\/TSE.2006.102 Yuming Zhou and Hareton Leung. 2006. Empirical analysis of object-oriented design metrics for predicting high and low severity faults. IEEE Trans. Softw. Eng. 32, 10 (Oct. 2006), 771--789. DOI:https:\/\/doi.org\/10.1109\/TSE.2006.102","journal-title":"IEEE Trans. Softw. Eng."},{"key":"e_1_2_1_119_1","volume-title":"Proceedings of the ACM\/IEEE 30th International Conference on Software Engineering. 531--540","author":"Zimmermann T.","unstructured":"T. Zimmermann and N. Nagappan . 2008. Predicting defects using network analysis on dependency graphs . In Proceedings of the ACM\/IEEE 30th International Conference on Software Engineering. 531--540 . DOI:https:\/\/doi.org\/10.1145\/1368088.1368161 10.1145\/1368088.1368161 T. Zimmermann and N. Nagappan. 2008. Predicting defects using network analysis on dependency graphs. In Proceedings of the ACM\/IEEE 30th International Conference on Software Engineering. 531--540. DOI:https:\/\/doi.org\/10.1145\/1368088.1368161"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3408896","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3408896","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:47:58Z","timestamp":1750178878000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3408896"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,9,26]]},"references-count":117,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,10,31]]}},"alternative-id":["10.1145\/3408896"],"URL":"https:\/\/doi.org\/10.1145\/3408896","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,9,26]]},"assertion":[{"value":"2019-09-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-06-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-09-26","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}