{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,11]],"date-time":"2026-04-11T02:12:43Z","timestamp":1775873563104,"version":"3.50.1"},"reference-count":48,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2024,4,17]],"date-time":"2024-04-17T00:00:00Z","timestamp":1713312000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62172205, 62072194, 62172202, 62272221, 62272037"],"award-info":[{"award-number":["62172205, 62072194, 62172202, 62272221, 62272037"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100004608","name":"Natural Science Foundation of Jiangsu Province","doi-asserted-by":"crossref","award":["BK20231402"],"award-info":[{"award-number":["BK20231402"]}],"id":[{"id":"10.13039\/501100004608","id-type":"DOI","asserted-by":"crossref"}]},{"name":"NJU-Huawei Software New Technology Joint Laboratory Fund","award":["TC20230202021-2023-08"],"award-info":[{"award-number":["TC20230202021-2023-08"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,5,31]]},"abstract":"<jats:p>\n            <jats:bold>Background.<\/jats:bold>\n            Software testing is a critical activity for ensuring the quality and reliability of software systems. To evaluate the effectiveness of different test suites, researchers have developed a variety of metrics.\n            <jats:bold>Problem.<\/jats:bold>\n            However, comparing these metrics is challenging due to the lack of a standardized evaluation framework including comprehensive factors. As a result, researchers often focus on single factors (e.g., size), which finally leads to different or even contradictory conclusions. After comparing dozens of pieces of work in detail, we have found two main problems most troubling to our community: (1) researchers tend to oversimplify the description of the ground truth they use, and (2) data involving real defects is not suitable for analysis using traditional statistical indicators.\n            <jats:bold>Objective.<\/jats:bold>\n            We aim at scrutinizing the whole process of comparing test suites for our community.\n            <jats:bold>Method.<\/jats:bold>\n            To hit this aim, we propose a framework ASSENT (ev\n            <jats:underline>\n              <jats:bold>A<\/jats:bold>\n            <\/jats:underline>\n            luating te\n            <jats:underline>\n              <jats:bold>S<\/jats:bold>\n            <\/jats:underline>\n            t\n            <jats:underline>\n              <jats:bold>S<\/jats:bold>\n            <\/jats:underline>\n            uite\n            <jats:underline>\n              <jats:bold>E<\/jats:bold>\n            <\/jats:underline>\n            ffective\n            <jats:underline>\n              <jats:bold>N<\/jats:bold>\n            <\/jats:underline>\n            ess me\n            <jats:underline>\n              <jats:bold>T<\/jats:bold>\n            <\/jats:underline>\n            rics) to guide the follow-up research for evaluating a test suite effectiveness metric. ASSENT consists of three fundamental components: ground truth, benchmark test suites, and agreement indicator. Its functioning is as follows: first, users clarify the ground truth for determining the real order in effectiveness among test suites. Second, users generate a set of benchmark test suites and derive their ground truth order in effectiveness. Third, users use the metric to derive the order in effectiveness for the same test suites. Finally, users calculate the agreement indicator between the two orders derived by two metrics.\n            <jats:bold>Result.<\/jats:bold>\n            With ASSENT, we are able to compare the accuracy of different test suite effectiveness metrics. We apply ASSENT to evaluate representative test suite effectiveness metrics, including mutation score and code coverage metrics. Our results show that, based on the real faults, mutation score, and subsuming mutation score are the best metrics to quantify test suite effectiveness. Meanwhile, by using mutants instead of real faults, test effectiveness will be overestimated by more than 20% in values.\n            <jats:bold>Conclusion.<\/jats:bold>\n            We recommend that the standardized evaluation framework ASSENT should be used for evaluating and comparing test effectiveness metrics in the future work.\n          <\/jats:p>","DOI":"10.1145\/3635713","type":"journal-article","created":{"date-parts":[[2023,12,5]],"date-time":"2023-12-05T12:02:39Z","timestamp":1701777759000},"page":"1-32","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Assessing Effectiveness of Test Suites: What Do We Know and What Should We Do?"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2902-2249","authenticated-orcid":false,"given":"Peng","family":"Zhang","sequence":"first","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China and Huawei Technologies Co., Ltd, Hangzhou, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6771-2139","authenticated-orcid":false,"given":"Yang","family":"Wang","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3831-5505","authenticated-orcid":false,"given":"Xutong","family":"Liu","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-7370-6421","authenticated-orcid":false,"given":"Zeyu","family":"Lu","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1153-2013","authenticated-orcid":false,"given":"Yibiao","family":"Yang","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2282-7175","authenticated-orcid":false,"given":"Yanhui","family":"Li","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2352-2226","authenticated-orcid":false,"given":"Lin","family":"Chen","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0494-5285","authenticated-orcid":false,"given":"Ziyuan","family":"Wang","sequence":"additional","affiliation":[{"name":"Nanjing University of Posts and Telecommunications, Nanjing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3696-6176","authenticated-orcid":false,"given":"Chang-Ai","family":"Sun","sequence":"additional","affiliation":[{"name":"University of Science and Technology Beijing, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4473-3068","authenticated-orcid":false,"given":"Xiao","family":"Yu","sequence":"additional","affiliation":[{"name":"Huawei Technologies Co., Ltd, Hangzhou, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4645-2526","authenticated-orcid":false,"given":"Yuming","family":"Zhou","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}]}],"member":"320","published-online":{"date-parts":[[2024,4,17]]},"reference":[{"key":"e_1_3_3_2_2","unstructured":"[n. d.]. https:\/\/github.com\/zhangpengNJU\/ASSENT\/README.md. Accessed on: 2023-09-25."},{"key":"e_1_3_3_3_2","unstructured":"[n. d.]. https:\/\/github.com\/cobertura. Accessed on: 2023-09-25."},{"key":"e_1_3_3_4_2","doi-asserted-by":"crossref","first-page":"547","DOI":"10.1145\/2950290.2950324","volume-title":"Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering","author":"Ahmed Iftekhar","year":"2016","unstructured":"Iftekhar Ahmed, Rahul Gopinath, Caius Brindescu, Alex Groce, and Carlos Jensen. 2016. Can testedness be effectively measured?. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. 547\u2013558."},{"key":"e_1_3_3_5_2","first-page":"21","volume-title":"Proceedings of the 2014 IEEE 7th International Conference on Software Testing, Verification, and Validation","author":"Ammann Paul","year":"2014","unstructured":"Paul Ammann, Marcio Eduardo Delamaro, and Jeff Offutt. 2014. Establishing theoretical minimal sets of mutants. In Proceedings of the 2014 IEEE 7th International Conference on Software Testing, Verification, and Validation. IEEE, 21\u201330."},{"key":"e_1_3_3_6_2","first-page":"1","volume-title":"Proceedings of the International Symposium on Formal Methods for Components and Objects","author":"Ball Thomas","year":"2004","unstructured":"Thomas Ball. 2004. A theory of predicate-complete test coverage and generation. In Proceedings of the International Symposium on Formal Methods for Components and Objects. Springer, 1\u201322."},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/1134285.1134299"},{"key":"e_1_3_3_8_2","volume-title":"Coverage-based Testing Strategies and Reliability Modeling for Fault-tolerant Software Systems","author":"Cai Xia","year":"2006","unstructured":"Xia Cai. 2006. Coverage-based Testing Strategies and Reliability Modeling for Fault-tolerant Software Systems. The Chinese University of Hong Kong (People\u2019s Republic of China)."},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/1083274.1083288"},{"key":"e_1_3_3_10_2","first-page":"597","volume-title":"Proceedings of the 2017 IEEE\/ACM 39th International Conference on Software Engineering (ICSE \u201917)","author":"Chekam Thierry Titcheu","year":"2017","unstructured":"Thierry Titcheu Chekam, Mike Papadakis, Yves Le Traon, and Mark Harman. 2017. An empirical study on mutation, statement and branch coverage fault revelation that avoids the unreliable clean program assumption. In Proceedings of the 2017 IEEE\/ACM 39th International Conference on Software Engineering (ICSE \u201917). IEEE, 597\u2013608."},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416667"},{"key":"e_1_3_3_12_2","first-page":"34","volume-title":"Proceedings of the 2009 IEEE 31st International Conference on Software Engineering","author":"Chilimbi Trishul M.","year":"2009","unstructured":"Trishul M. Chilimbi, Ben Liblit, Krishna Mehra, Aditya V. Nori, and Kapil Vaswani. 2009. Holmes: Effective statistical debugging via efficient path profiling. In Proceedings of the 2009 IEEE 31st International Conference on Software Engineering. IEEE, 34\u201344."},{"key":"e_1_3_3_13_2","doi-asserted-by":"crossref","unstructured":"Jacob Cohen. 1992. Quantitative methods in psychology: A power primer. Psychol. Bull. 112 1 (1992) 1155\u20131159.","DOI":"10.1037\/\/0033-2909.112.1.155"},{"key":"e_1_3_3_14_2","first-page":"203","volume-title":"Proceedings of the 2014 IEEE 7th International Conference on Software Testing, Verification, and Validation","author":"Delamaro Marcio Eduardo","year":"2014","unstructured":"Marcio Eduardo Delamaro, Lin Deng, Vinicius Humberto Serapilha Durelli, Nan Li, and Jeff Offutt. 2014. Experimental evaluation of SDL and one-op mutation for C. In Proceedings of the 2014 IEEE 7th International Conference on Software Testing, Verification, and Validation. IEEE, 203\u2013212."},{"key":"e_1_3_3_15_2","first-page":"11","volume-title":"Proceedings of the 2014 IEEE 7th International Conference on Software Testing, Verification, and Validation","author":"Delamaro Marcio Eduardo","year":"2014","unstructured":"Marcio Eduardo Delamaro, Jeff Offutt, and Paul Ammann. 2014. Designing deletion mutation operators. In Proceedings of the 2014 IEEE 7th International Conference on Software Testing, Verification, and Validation. IEEE, 11\u201320."},{"issue":"4","key":"e_1_3_3_16_2","doi-asserted-by":"crossref","first-page":"e1630","DOI":"10.1002\/stvr.1630","article-title":"Assessment of C++ object-oriented mutation operators: A selective mutation approach","volume":"27","author":"Delgado-P\u00e9rez Pedro","year":"2017","unstructured":"Pedro Delgado-P\u00e9rez, Sergio Segura, and Inmaculada Medina-Bulo. 2017. Assessment of C++ object-oriented mutation operators: A selective mutation approach. Software Testing, Verification and Reliability 27, 4-5 (2017), e1630.","journal-title":"Software Testing, Verification and Reliability"},{"issue":"4","key":"e_1_3_3_17_2","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1109\/C-M.1978.218136","article-title":"Hints on test data selection: Help for the practicing programmer","volume":"11","author":"DeMillo Richard A.","year":"1978","unstructured":"Richard A. DeMillo, Richard J. Lipton, and Frederick G. Sayward. 1978. Hints on test data selection: Help for the practicing programmer. Computer 11, 4 (1978), 34\u201341.","journal-title":"Computer"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/2660767"},{"key":"e_1_3_3_19_2","doi-asserted-by":"crossref","unstructured":"Dunwei Gong Gongjie Zhang Xiangjuan Yao and Fanlin Meng. 2017. Mutant reduction based on dominance relation for weak mutation testing. Information and Software Technology 81 C (2017) 82\u201396.","DOI":"10.1016\/j.infsof.2016.05.001"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568278"},{"issue":"4","key":"e_1_3_3_21_2","doi-asserted-by":"crossref","first-page":"758","DOI":"10.1109\/TSE.2019.2903057","article-title":"Lightweight assessment of test-case effectiveness using source-code-quality indicators","volume":"47","author":"Grano Giovanni","year":"2019","unstructured":"Giovanni Grano, Fabio Palomba, and Harald C. Gall. 2019. Lightweight assessment of test-case effectiveness using source-code-quality indicators. IEEE Transactions on Software Engineering 47, 4 (2019), 758\u2013774.","journal-title":"IEEE Transactions on Software Engineering"},{"issue":"4","key":"e_1_3_3_22_2","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1109\/TSE.1977.231145","article-title":"Testing programs with the aid of a compiler","author":"Hamlet Richard G.","year":"1977","unstructured":"Richard G. Hamlet. 1977. Testing programs with the aid of a compiler. IEEE Transactions on Software Engineering4 (1977), 279\u2013290.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_3_23_2","doi-asserted-by":"crossref","first-page":"114","DOI":"10.1109\/ICST.2019.00021","volume-title":"Proceedings of the 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST \u201919)","author":"Hariri Farah","year":"2019","unstructured":"Farah Hariri, August Shi, Vimuth Fernando, Suleman Mahmood, and Darko Marinov. 2019. Comparing mutation testing at the levels of source code and compiler intermediate representation. In Proceedings of the 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST \u201919). IEEE, 114\u2013124."},{"key":"e_1_3_3_24_2","first-page":"9","article-title":"Mutation clustering","author":"Hussain Shamaila","year":"2008","unstructured":"Shamaila Hussain. 2008. Mutation clustering. Ms. Th., Kings College London, Strand, London (2008), 9.","journal-title":"Ms. Th., Kings College London, Strand, London"},{"key":"e_1_3_3_25_2","doi-asserted-by":"crossref","first-page":"435","DOI":"10.1145\/2568225.2568271","volume-title":"Proceedings of the 36th International Conference on Software Engineering","author":"Inozemtseva Laura","year":"2014","unstructured":"Laura Inozemtseva and Reid Holmes. 2014. Coverage is not strongly correlated with test suite effectiveness. In Proceedings of the 36th International Conference on Software Engineering. 435\u2013445."},{"key":"e_1_3_3_26_2","first-page":"42","volume-title":"2012 1st International Workshop on Realizing AI Synergies in Software Engineering (RAISE \u201912)","author":"Jalbert Kevin","year":"2012","unstructured":"Kevin Jalbert and Jeremy S. Bradbury. 2012. Predicting mutation score using source code and test suite metrics. In 2012 1st International Workshop on Realizing AI Synergies in Software Engineering (RAISE \u201912). IEEE, 42\u201346."},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2010.62"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635929"},{"key":"e_1_3_3_29_2","doi-asserted-by":"crossref","first-page":"612","DOI":"10.1109\/ASE.2011.6100138","volume-title":"Proceedings of the 2011 26th IEEE\/ACM International Conference on Automated Software Engineering (ASE \u201911)","author":"Just Ren\u00e9","year":"2011","unstructured":"Ren\u00e9 Just, Franz Schweiggert, and Gregory M. Kapfhammer. 2011. MAJOR: An efficient and extensible tool for mutation analysis in a Java compiler. In Proceedings of the 2011 26th IEEE\/ACM International Conference on Automated Software Engineering (ASE \u201911). IEEE, 612\u2013615."},{"key":"e_1_3_3_30_2","first-page":"560","volume-title":"Proceedings of the 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER \u201915)","author":"Kochhar Pavneet Singh","year":"2015","unstructured":"Pavneet Singh Kochhar, Ferdian Thung, and David Lo. 2015. Code coverage and test suite effectiveness: Empirical study with real bugs in large systems. In Proceedings of the 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER \u201915). IEEE, 560\u2013564."},{"key":"e_1_3_3_31_2","first-page":"176","volume-title":"Proceedings of the 2014 IEEE 7th International Conference on Software Testing, Verification, and Validation Workshops","author":"Kurtz Bob","year":"2014","unstructured":"Bob Kurtz, Paul Ammann, Marcio E. Delamaro, Jeff Offutt, and Lin Deng. 2014. Mutant subsumption graphs. In Proceedings of the 2014 IEEE 7th International Conference on Software Testing, Verification, and Validation Workshops. IEEE, 176\u2013185."},{"key":"e_1_3_3_32_2","first-page":"1","volume-title":"Proceedings of the 2015 IEEE 8th International Conference on Software Testing, Verification, and Validation Workshops (ICSTW \u201915)","author":"Kurtz Bob","year":"2015","unstructured":"Bob Kurtz, Paul Ammann, and Jeff Offutt. 2015. Static analysis of mutant subsumption. In Proceedings of the 2015 IEEE 8th International Conference on Software Testing, Verification, and Validation Workshops (ICSTW \u201915). IEEE, 1\u201310."},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/2950290.2950322"},{"issue":"5","key":"e_1_3_3_34_2","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1145\/301631.301678","article-title":"Whole program paths","volume":"34","author":"Larus James R.","year":"1999","unstructured":"James R. Larus. 1999. Whole program paths. ACM SIGPLAN Notices 34, 5 (1999), 259\u2013269.","journal-title":"ACM SIGPLAN Notices"},{"key":"e_1_3_3_35_2","first-page":"604","volume-title":"1991 Proceedings of the 15th Annual International Computer Software and Applications Conference","author":"Mathur Aditya P.","year":"1991","unstructured":"Aditya P. Mathur. 1991. Performance, effectiveness, and reliability issues in software testing. In 1991 Proceedings of the 15th Annual International Computer Software and Applications Conference. IEEE Computer Society, 604\u2013605."},{"key":"e_1_3_3_36_2","first-page":"57","volume-title":"Proceedings of the 18th International Symposium on Software Testing and Analysis","author":"Namin Akbar Siami","year":"2009","unstructured":"Akbar Siami Namin and James H. Andrews. 2009. The influence of size and coverage on test suite effectiveness. In Proceedings of the 18th International Symposium on Software Testing and Analysis. 57\u201368."},{"issue":"2","key":"e_1_3_3_37_2","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1145\/227607.227610","article-title":"An experimental determination of sufficient mutant operators","volume":"5","author":"Offutt A. Jefferson","year":"1996","unstructured":"A. Jefferson Offutt, Ammei Lee, Gregg Rothermel, Roland H. Untch, and Christian Zapf. 1996. An experimental determination of sufficient mutant operators. ACM Transactions on Software Engineering and Methodology 5, 2 (1996), 99\u2013118.","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"key":"e_1_3_3_38_2","first-page":"275","volume-title":"Proceedings of the Advances in Computers","author":"Papadakis Mike","year":"2019","unstructured":"Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Mutation testing advances: An analysis and survey. In Proceedings of the Advances in Computers. Elsevier, 275\u2013378."},{"key":"e_1_3_3_39_2","doi-asserted-by":"crossref","first-page":"90","DOI":"10.1109\/ICSTW.2010.50","volume-title":"Proceedings of the 2010 3rd International Conference on Software Testing, Verification, and Validation Workshops","author":"Papadakis Mike","year":"2010","unstructured":"Mike Papadakis and Nicos Malevris. 2010. An empirical evaluation of the first and second order mutation testing strategies. In Proceedings of the 2010 3rd International Conference on Software Testing, Verification, and Validation Workshops. IEEE, 90\u201399."},{"key":"e_1_3_3_40_2","doi-asserted-by":"crossref","first-page":"537","DOI":"10.1145\/3180155.3180183","volume-title":"Proceedings of the 40th International Conference on Software Engineering","author":"Papadakis Mike","year":"2018","unstructured":"Mike Papadakis, Donghwan Shin, Shin Yoo, and Doo-Hwan Bae. 2018. Are mutation scores correlated with real fault detection? A large scale empirical study on the relationship between mutants and real faults. In Proceedings of the 40th International Conference on Software Engineering. 537\u2013548."},{"key":"e_1_3_3_41_2","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1109\/MSR.2019.00028","volume-title":"Proceedings of the 2019 IEEE\/ACM 16th International Conference on Mining Software Repositories (MSR \u201919)","author":"Serra Domenico","year":"2019","unstructured":"Domenico Serra, Giovanni Grano, Fabio Palomba, Filomena Ferrucci, Harald C. Gall, and Alberto Bacchelli. 2019. On the effectiveness of manual and automatic unit test generation: Ten years later. In Proceedings of the 2019 IEEE\/ACM 16th International Conference on Mining Software Repositories (MSR \u201919). IEEE, 121\u2013125."},{"issue":"10","key":"e_1_3_3_42_2","doi-asserted-by":"crossref","first-page":"914","DOI":"10.1109\/TSE.2017.2732347","article-title":"A theoretical and empirical study of diversity-aware mutation adequacy criterion","volume":"44","author":"Shin Donghwan","year":"2017","unstructured":"Donghwan Shin, Shin Yoo, and Doo-Hwan Bae. 2017. A theoretical and empirical study of diversity-aware mutation adequacy criterion. IEEE Transactions on Software Engineering 44, 10 (2017), 914\u2013931.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_3_43_2","first-page":"351","volume-title":"Proceedings of the 30th International Conference on Software Engineering","author":"Namin Akbar Siami","year":"2008","unstructured":"Akbar Siami Namin, James H. Andrews, and Duncan J. Murdoch. 2008. Sufficient mutation operators for measuring test effectiveness. In Proceedings of the 30th International Conference on Software Engineering. 351\u2013360."},{"key":"e_1_3_3_44_2","first-page":"1","volume-title":"Proceedings of the 37th IEEE\/ACM International Conference on Automated Software Engineering","author":"Tian Zhao","year":"2022","unstructured":"Zhao Tian, Junjie Chen, Qihao Zhu, Junjie Yang, and Lingming Zhang. 2022. Learning to construct better mutation faults. In Proceedings of the 37th IEEE\/ACM International Conference on Automated Software Engineering. 1\u201313."},{"key":"e_1_3_3_45_2","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1109\/METRIC.1993.263796","volume-title":"[1993] Proceedings of the 1st International Software Metrics Symposium","author":"Weyuker Elaine J.","year":"1993","unstructured":"Elaine J. Weyuker. 1993. Can we measure software testing effectiveness?. In [1993] Proceedings of the 1st International Software Metrics Symposium. IEEE, 100\u2013107."},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1016\/0164-1212(94)00098-0"},{"issue":"7","key":"e_1_3_3_47_2","doi-asserted-by":"crossref","first-page":"433","DOI":"10.1016\/j.infsof.2005.05.003","article-title":"On the relationship between two control-flow coverage criteria: All JJ-paths and MCDC","volume":"48","author":"Woodward Martin R.","year":"2006","unstructured":"Martin R. Woodward and Michael A. Hennell. 2006. On the relationship between two control-flow coverage criteria: All JJ-paths and MCDC. Information and Software Technology 48, 7 (2006), 433\u2013440.","journal-title":"Information and Software Technology"},{"issue":"3","key":"e_1_3_3_48_2","doi-asserted-by":"crossref","first-page":"1067","DOI":"10.1109\/TSE.2020.3010361","article-title":"Cbua: A probabilistic, predictive, and practical approach for evaluating test suite effectiveness","volume":"48","author":"Zhang Peng","year":"2020","unstructured":"Peng Zhang, Yanhui Li, Wanwangying Ma, Yibiao Yang, Lin Chen, Hongmin Lu, Yuming Zhou, and Baowen Xu. 2020. Cbua: A probabilistic, predictive, and practical approach for evaluating test suite effectiveness. IEEE Transactions on Software Engineering 48, 3 (2020), 1067\u20131096.","journal-title":"IEEE Transactions on Software Engineering"},{"issue":"4","key":"e_1_3_3_49_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3522578","article-title":"Mutant reduction evaluation: What is there and what is missing?","volume":"31","author":"Zhang Peng","year":"2022","unstructured":"Peng Zhang, Yang Wang, Xutong Liu, Yanhui Li, Yibiao Yang, Ziyuan Wang, Xiaoyu Zhou, Lin Chen, and Yuming Zhou. 2022. Mutant reduction evaluation: What is there and what is missing? ACM Transactions on Software Engineering and Methodology 31, 4 (2022), 1\u201346.","journal-title":"ACM Transactions on Software Engineering and Methodology"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3635713","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3635713","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T23:56:59Z","timestamp":1750291019000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3635713"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,17]]},"references-count":48,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,5,31]]}},"alternative-id":["10.1145\/3635713"],"URL":"https:\/\/doi.org\/10.1145\/3635713","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,17]]},"assertion":[{"value":"2023-04-30","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-11-22","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-04-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}