{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,5]],"date-time":"2026-02-05T11:21:35Z","timestamp":1770290495464,"version":"3.49.0"},"reference-count":69,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2020,6,1]],"date-time":"2020-06-01T00:00:00Z","timestamp":1590969600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2020,7,31]]},"abstract":"<jats:p>Coverage criteria provide a useful and widely used means to guide software testing; however, indiscriminately pursuing full coverage may not always be convenient or meaningful, as not all entities are of interest in any usage context. We aim at introducing a more meaningful notion of coverage that takes into account how the software is going to be used. Entities that are not going to be exercised by the user should not contribute to the coverage ratio. We revisit the definition of coverage measures, introducing a notion of relative coverage. According to this notion, we provide a definition and a theoretical framework of relative coverage, within which we discuss implications on testing theory and practice. Through the evaluation of three different instances of relative coverage, we could observe that relative coverage measures provide a more effective strategy than traditional ones: we could reach higher coverage measures, and test cases selected by relative coverage could achieve higher reliability. We hint at several other useful implications of relative coverage notion on different aspects of software testing.<\/jats:p>","DOI":"10.1145\/3389126","type":"journal-article","created":{"date-parts":[[2020,6,1]],"date-time":"2020-06-01T16:19:41Z","timestamp":1591028381000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["Testing Relative to Usage Scope"],"prefix":"10.1145","volume":"29","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9608-9393","authenticated-orcid":false,"given":"Breno","family":"Miranda","sequence":"first","affiliation":[{"name":"Federal University of Pernambuco, Recife, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8749-1356","authenticated-orcid":false,"given":"Antonia","family":"Bertolino","sequence":"additional","affiliation":[{"name":"ISTI - CNR, Pisa, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,6]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-23716-4_6"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/381694.378832"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2014.2372785"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2010.10.024"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2017.56"},{"key":"e_1_2_1_6_1","first-page":"2906187","article-title":"Adaptive test case allocation, selection and generation using coverage spectrum and operational profile","volume":"2019","author":"Bertolino A.","year":"2019","unstructured":"A. Bertolino , B. Miranda , R. Pietrantuono , and S. Russo . 2019 . Adaptive test case allocation, selection and generation using coverage spectrum and operational profile . IEEE Transactions on Software Engineering. Early access. DOI:https:\/\/doi.org\/10.1109\/TSE. 2019 . 2906187 10.1109\/TSE.2019.2906187 A. Bertolino, B. Miranda, R. Pietrantuono, and S. Russo. 2019. Adaptive test case allocation, selection and generation using coverage spectrum and operational profile. IEEE Transactions on Software Engineering. Early access. DOI:https:\/\/doi.org\/10.1109\/TSE.2019.2906187","journal-title":"DOI:https:\/\/doi.org\/10.1109\/TSE."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.485220"},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the 28th International Conference on Software Engineering (ICSE\u201906)","author":"Briand Lionel C.","unstructured":"Lionel C. Briand , Yvan Labiche , and Michal M. S\u00f3wka . 2006. Automated, contract-based user testing of commercial-off-the-shelf components . In Proceedings of the 28th International Conference on Software Engineering (ICSE\u201906) . ACM, New York, NY, 92--101. DOI:https:\/\/doi.org\/10.1145\/1134285.1134300 10.1145\/1134285.1134300 Lionel C. Briand, Yvan Labiche, and Michal M. S\u00f3wka. 2006. Automated, contract-based user testing of commercial-off-the-shelf components. In Proceedings of the 28th International Conference on Software Engineering (ICSE\u201906). ACM, New York, NY, 92--101. DOI:https:\/\/doi.org\/10.1145\/1134285.1134300"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/11901914_30"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2408776.2408795"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0950-5849(98)00086-X"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238214"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/1147249.1147257"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/SRDS.2009.26"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/SCAM.2001.972675"},{"key":"e_1_2_1_16_1","volume-title":"Travassos","author":"Dias Neto Arilo C.","year":"2007","unstructured":"Arilo C. Dias Neto , Rajesh Subramanyan , Marlon Vieira , and Guilherme H . Travassos . 2007 . A survey on model-based testing approaches: A systematic review. In Proceedings of the 1st ACM International Workshop on Empirical Assessment of Software Engineering Languages, and Technologies: Held in Conjunction with the 22nd IEEE\/ACM International Conference on Automated Software Engineering (ASE\u201907). ACM, New York, NY , 31--36. Arilo C. Dias Neto, Rajesh Subramanyan, Marlon Vieira, and Guilherme H. Travassos. 2007. A survey on model-based testing approaches: A systematic review. In Proceedings of the 1st ACM International Workshop on Empirical Assessment of Software Engineering Languages, and Technologies: Held in Conjunction with the 22nd IEEE\/ACM International Conference on Automated Software Engineering (ASE\u201907). ACM, New York, NY, 31--36."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10009-014-0330-5"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11219-013-9226-y"},{"key":"e_1_2_1_19_1","first-page":"5","article-title":"A taxonomy of risk-based testing","volume":"16","author":"Felderer Michael","year":"2014","unstructured":"Michael Felderer and Ina Schieferdecker . 2014 . A taxonomy of risk-based testing . International Journal on Software Tools for Technology Transfer 16 , 5 (Oct. 2014), 559--568. DOI:https:\/\/doi.org\/10.1007\/s10009-014-0332-3 10.1007\/s10009-014-0332-3 Michael Felderer and Ina Schieferdecker. 2014. A taxonomy of risk-based testing. International Journal on Software Tools for Technology Transfer 16, 5 (Oct. 2014), 559--568. DOI:https:\/\/doi.org\/10.1007\/s10009-014-0332-3","journal-title":"International Journal on Software Tools for Technology Transfer"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.707695"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.6194"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.221133"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3194718.3194719"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-01973-9_78"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1065010.1065036"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1975.6312836"},{"key":"e_1_2_1_27_1","first-page":"44","article-title":"Toward a theory of testing: Data selection criteria","volume":"2","author":"Goodenough J. B.","year":"1977","unstructured":"J. B. Goodenough and S. L. Gerhart . 1977 . Toward a theory of testing: Data selection criteria . Current Trends in Programming Methodology 2 , 2 (1977), 44 -- 79 . J. B. Goodenough and S. L. Gerhart. 1977. Toward a theory of testing: Data selection criteria. Current Trends in Programming Methodology 2, 2 (1977), 44--79.","journal-title":"Current Trends in Programming Methodology"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1983.235433"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF00873408"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/277631.277647"},{"key":"e_1_2_1_31_1","first-page":"3","article-title":"Reliability of the path analysis testing strategy","volume":"2","author":"Howden W. E.","year":"1976","unstructured":"W. E. Howden . 1976 . Reliability of the path analysis testing strategy . IEEE Transactions on Software Engineering 2 , 3 (May 1976), 208--215. DOI:https:\/\/doi.org\/10.1109\/TSE.1976.233816 10.1109\/TSE.1976.233816 W. E. Howden. 1976. Reliability of the path analysis testing strategy. IEEE Transactions on Software Engineering 2, 3 (May 1976), 208--215. DOI:https:\/\/doi.org\/10.1109\/TSE.1976.233816","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2931037.2931062"},{"key":"e_1_2_1_33_1","volume-title":"Proceedings of the USENIX Annual Technical Conference. 243--254","author":"Kasikci Baris","year":"2014","unstructured":"Baris Kasikci , Thomas Ball , George Candea , John Erickson , and Madanlal Musuvathi . 2014 . Efficient tracing of cold code via bias-free sampling . In Proceedings of the USENIX Annual Technical Conference. 243--254 . Baris Kasikci, Thomas Ball, George Candea, John Erickson, and Madanlal Musuvathi. 2014. Efficient tracing of cold code via bias-free sampling. In Proceedings of the USENIX Annual Technical Conference. 243--254."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1016\/0020-0190(88)90054-3"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2007.41"},{"key":"e_1_2_1_36_1","volume-title":"Measurement, Modelling, and Evaluation of Computing Systems and Dependability and Fault Tolerance","author":"Meitner Matthias","unstructured":"Matthias Meitner and Francesca Saglietti . 2014. Target-specific adaptations of coupling-based software reliability testing . In Measurement, Modelling, and Evaluation of Computing Systems and Dependability and Fault Tolerance . Springer , 192--206. Matthias Meitner and Francesca Saglietti. 2014. Target-specific adaptations of coupling-based software reliability testing. In Measurement, Modelling, and Evaluation of Computing Systems and Dependability and Fault Tolerance. Springer, 192--206."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642937.2653471"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2593501.2593505"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/SEAA.2015.69"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2896921.2896934"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2016.06.058"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11219-017-9388-0"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/52.199724"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/2.544239"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/566172.566182"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2950290.2950336"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/SEAA.2012.43"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/859670.859678"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/ECBS.2001.922409"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0164-1212(99)00084-9"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0950-5849(98)00093-7"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-22183-0_7"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/248233.248262"},{"key":"e_1_2_1_56_1","first-page":"195","article-title":"Statistical usage testing for software reliability control","volume":"19","author":"Runeson Per","year":"1995","unstructured":"Per Runeson and Claes Wohlin . 1995 . Statistical usage testing for software reliability control . Informatica 19 , 2 (1995), 195 -- 207 . Per Runeson and Claes Wohlin. 1995. Statistical usage testing for software reliability control. Informatica 19, 2 (1995), 195--207.","journal-title":"Informatica"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/1095430.1081750"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/2518106"},{"key":"e_1_2_1_59_1","series-title":"Lecture Notes in Computer Science","volume-title":"Fundamental Approaches to Software Engineering","author":"Staats Matt","unstructured":"Matt Staats , Gregory Gay , Michael Whalen , and Mats Heimdahl . 2012. On the danger of coverage directed test case generation . In Fundamental Approaches to Software Engineering . Lecture Notes in Computer Science , Vol. 7212 . Springer , 409--424. DOI:https:\/\/doi.org\/10.1007\/978-3-642-28872-2_28 10.1007\/978-3-642-28872-2_28 Matt Staats, Gregory Gay, Michael Whalen, and Mats Heimdahl. 2012. On the danger of coverage directed test case generation. In Fundamental Approaches to Software Engineering. Lecture Notes in Computer Science, Vol. 7212. Springer, 409--424. DOI:https:\/\/doi.org\/10.1007\/978-3-642-28872-2_28"},{"key":"e_1_2_1_60_1","volume-title":"Proceedings of the 33rd International Conference on Software Engineering. ACM","author":"Staats Matt","unstructured":"Matt Staats , Michael W. Whalen , and Mats P. E. Heimdahl . 2011. Programs, tests, and oracles: The foundations of testing revisited . In Proceedings of the 33rd International Conference on Software Engineering. ACM , New York, NY, 391--400. Matt Staats, Michael W. Whalen, and Mats P. E. Heimdahl. 2011. Programs, tests, and oracles: The foundations of testing revisited. In Proceedings of the 33rd International Conference on Software Engineering. ACM, New York, NY, 391--400."},{"key":"e_1_2_1_61_1","volume-title":"Artificial Intelligence, Networking and Parallel\/Distributed Computing","author":"Takagi Tomohiko","unstructured":"Tomohiko Takagi , Kazuya Nishimachi , Masayuki Muragishi , Takashi Mitsuhashi , and Zengo Furukawa . 2009. Usage distribution coverage: What percentage of expected use has been executed in software testing? In Software Engineering , Artificial Intelligence, Networking and Parallel\/Distributed Computing . Springer , 57--67. Tomohiko Takagi, Kazuya Nishimachi, Masayuki Muragishi, Takashi Mitsuhashi, and Zengo Furukawa. 2009. Usage distribution coverage: What percentage of expected use has been executed in software testing? In Software Engineering, Artificial Intelligence, Networking and Parallel\/Distributed Computing. Springer, 57--67."},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/566171.566186"},{"key":"e_1_2_1_63_1","volume-title":"Proceedings of the IEEE International Conference onSoftware Science, Technology, and Engineering (SwSTE\u201903)","author":"Torkar R.","year":"2003","unstructured":"R. Torkar and S. Mankefors . 2003. A survey on testing and reuse . In Proceedings of the IEEE International Conference onSoftware Science, Technology, and Engineering (SwSTE\u201903) . 164--173. DOI:https:\/\/doi.org\/10.1109\/SWSTE. 2003 .1245437 10.1109\/SWSTE.2003.1245437 R. Torkar and S. Mankefors. 2003. A survey on testing and reuse. In Proceedings of the IEEE International Conference onSoftware Science, Technology, and Engineering (SwSTE\u201903). 164--173. DOI:https:\/\/doi.org\/10.1109\/SWSTE.2003.1245437"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/113445.113455"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/52.73748"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.153381"},{"key":"e_1_2_1_67_1","unstructured":"Mark David Weiser. 1979. Program Slices: Formal Psychological and Practical Investigations of an Automatic Program Abstraction Method. Ph.D. Dissertation University of Michigan.  Mark David Weiser. 1979. Program Slices: Formal Psychological and Practical Investigations of an Automatic Program Abstraction Method. Ph.D. Dissertation University of Michigan."},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1109\/52.714817"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1980.234485"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/2786805.2786858"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/267580.267590"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3389126","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3389126","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:33:02Z","timestamp":1750199582000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3389126"}},"subtitle":["Revisiting Software Coverage Criteria"],"short-title":[],"issued":{"date-parts":[[2020,6]]},"references-count":69,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2020,7,31]]}},"alternative-id":["10.1145\/3389126"],"URL":"https:\/\/doi.org\/10.1145\/3389126","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,6]]},"assertion":[{"value":"2018-12-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-03-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-06-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}