{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,17]],"date-time":"2026-02-17T12:16:40Z","timestamp":1771330600906,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":43,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,4,15]],"date-time":"2020-04-15T00:00:00Z","timestamp":1586908800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,4,15]]},"DOI":"10.1145\/3383219.3383236","type":"proceedings-article","created":{"date-parts":[[2020,5,4]],"date-time":"2020-05-04T07:51:02Z","timestamp":1588578662000},"page":"160-169","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Which Software Faults Are Tests Not Detecting?"],"prefix":"10.1145","author":[{"given":"Jean","family":"Petri\u0107","sequence":"first","affiliation":[{"name":"Lancaster University Lancaster, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tracy","family":"Hall","sequence":"additional","affiliation":[{"name":"Lancaster University Lancaster, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"David","family":"Bowes","sequence":"additional","affiliation":[{"name":"Lancaster University Lancaster, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,4,17]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"[n.d.]. Mutation operators. http:\/\/pitest.org\/quickstart\/mutators\/. (Accessed on 02\/20\/2019).  [n.d.]. Mutation operators. http:\/\/pitest.org\/quickstart\/mutators\/. (Accessed on 02\/20\/2019)."},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2950290.2950324"},{"key":"e_1_3_2_1_3_1","unstructured":"Omar H Alhazmi Sung-Whan Woo and Yashwant K Malaiya. 2006. Security vulnerability categories in major software systems.. In Communication Network and Information Security. 138--143.  Omar H Alhazmi Sung-Whan Woo and Yashwant K Malaiya. 2006. Security vulnerability categories in major software systems.. In Communication Network and Information Security. 138--143."},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2013.02.061"},{"key":"e_1_3_2_1_5_1","volume-title":"Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005. 402--411","author":"Andrews J. H.","year":"2005","unstructured":"J. H. Andrews , L. C. Briand , and Y. Labiche . 2005. Is mutation an appropriate tool for testing experiments? [software testing] . In Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005. 402--411 . https:\/\/doi.org\/10.1109\/ICSE. 2005 .1553583 10.1109\/ICSE.2005.1553583 J. H. Andrews, L. C. Briand, and Y. Labiche. 2005. Is mutation an appropriate tool for testing experiments? [software testing]. In Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005. 402--411. https:\/\/doi.org\/10.1109\/ICSE.2005.1553583"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2017.3281318"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2014.2342227"},{"key":"e_1_3_2_1_8_1","volume-title":"Andrea De Lucia, and Dave Binkley","author":"Bavota Gabriele","year":"2015","unstructured":"Gabriele Bavota , Abdallah Qusef , Rocco Oliveto , Andrea De Lucia, and Dave Binkley . 2015 . Are test smells really harmful? An empirical study. Empirical Software Engineering 20, 4 (01 Aug 2015), 1052--1094. https:\/\/doi.org\/10.1007\/s10664-014-9313-0 10.1007\/s10664-014-9313-0 Gabriele Bavota, Abdallah Qusef, Rocco Oliveto, Andrea De Lucia, and Dave Binkley. 2015. Are test smells really harmful? An empirical study. Empirical Software Engineering 20, 4 (01 Aug 2015), 1052--1094. https:\/\/doi.org\/10.1007\/s10664-014-9313-0"},{"key":"#cr-split#-e_1_3_2_1_9_1.1","unstructured":"Hudson Silva Borges and Marco Tulio Valente. 2017. Application Domain of 5 000 GitHub Repositories. https:\/\/doi.org\/10.5281\/zenodo.804474 10.5281\/zenodo.804474"},{"key":"#cr-split#-e_1_3_2_1_9_1.2","unstructured":"Hudson Silva Borges and Marco Tulio Valente. 2017. Application Domain of 5 000 GitHub Repositories. https:\/\/doi.org\/10.5281\/zenodo.804474"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.5555\/3106039.3106042"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2931037.2948707"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.910859"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/ESEM.2013.50"},{"key":"e_1_3_2_1_14_1","volume-title":"Empirical Evaluation of the Statement Deletion Mutation Operator. In 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation. 84--93","author":"Deng L.","year":"2013","unstructured":"L. Deng , J. Offutt , and N. Li . 2013 . Empirical Evaluation of the Statement Deletion Mutation Operator. In 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation. 84--93 . https:\/\/doi.org\/10.1109\/ICST. 2013 .20 10.1109\/ICST.2013.20 L. Deng, J. Offutt, and N. Li. 2013. Empirical Evaluation of the Statement Deletion Mutation Operator. In 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation. 84--93. https:\/\/doi.org\/10.1109\/ICST.2013.20"},{"key":"e_1_3_2_1_15_1","first-page":"3","article-title":"Dynamic Selection of Classifiers in Bug Prediction","volume":"1","author":"Nucci D. Di","year":"2017","unstructured":"D. Di Nucci , F. Palomba , R. Oliveto , and A. De Lucia . 2017 . Dynamic Selection of Classifiers in Bug Prediction : An Adaptive Method. IEEE Transactions on Emerging Topics in Computational Intelligence 1 , 3 (June 2017), 202--212. https:\/\/doi.org\/10.1109\/TETCI.2017.2699224 10.1109\/TETCI.2017.2699224 D. Di Nucci, F. Palomba, R. Oliveto, and A. De Lucia. 2017. Dynamic Selection of Classifiers in Bug Prediction: An Adaptive Method. IEEE Transactions on Emerging Topics in Computational Intelligence 1, 3 (June 2017), 202--212. https:\/\/doi.org\/10.1109\/TETCI.2017.2699224","journal-title":"An Adaptive Method. IEEE Transactions on Emerging Topics in Computational Intelligence"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2014.40"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW.2018.00023"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.5555\/3220900.3221112"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568271"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635929"},{"key":"e_1_3_2_1_21_1","volume-title":"How effective are mutation testing tools? An empirical analysis of Java mutation testing tools with manual analysis and real faults. Empirical Software Engineering 23, 4 (01","author":"Kintis Marinos","year":"2018","unstructured":"Marinos Kintis , Mike Papadakis , Andreas Papadopoulos , Evangelos Valvis , Nicos Malevris , and Yves Le Traon . 2018. How effective are mutation testing tools? An empirical analysis of Java mutation testing tools with manual analysis and real faults. Empirical Software Engineering 23, 4 (01 Aug 2018 ), 2426--2463. https:\/\/doi.org\/10.1007\/s10664-017-9582-5 10.1007\/s10664-017-9582-5 Marinos Kintis, Mike Papadakis, Andreas Papadopoulos, Evangelos Valvis, Nicos Malevris, and Yves Le Traon. 2018. How effective are mutation testing tools? An empirical analysis of Java mutation testing tools with manual analysis and real faults. Empirical Software Engineering 23, 4 (01 Aug 2018), 2426--2463. https:\/\/doi.org\/10.1007\/s10664-017-9582-5"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/TR.2017.2727062"},{"key":"e_1_3_2_1_23_1","volume-title":"How Redundant and Equivalent Mutants Affect Determination of Test Completeness. In 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW). 142--151","author":"Kurtz B.","year":"2016","unstructured":"B. Kurtz , P. Ammann , J. Offutt , and M. Kurtz . 2016. Are We There Yet? How Redundant and Equivalent Mutants Affect Determination of Test Completeness. In 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW). 142--151 . https:\/\/doi.org\/10.1109\/ICSTW. 2016 .41 10.1109\/ICSTW.2016.41 B. Kurtz, P. Ammann, J. Offutt, and M. Kurtz. 2016. Are We There Yet? How Redundant and Equivalent Mutants Affect Determination of Test Completeness. In 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW). 142--151. https:\/\/doi.org\/10.1109\/ICSTW.2016.41"},{"key":"e_1_3_2_1_24_1","volume-title":"Multi Objective Higher Order Mutation Testing with Genetic Programming. In 2009 Testing: Academic and Industrial Conference -Practice and Research Techniques. 21--29","author":"Langdon W. B.","year":"2009","unstructured":"W. B. Langdon , M. Harman , and Y. Jia . 2009 . Multi Objective Higher Order Mutation Testing with Genetic Programming. In 2009 Testing: Academic and Industrial Conference -Practice and Research Techniques. 21--29 . https:\/\/doi.org\/10.1109\/TAICPART. 2009 .18 10.1109\/TAICPART.2009.18 W. B. Langdon, M. Harman, and Y. Jia. 2009. Multi Objective Higher Order Mutation Testing with Genetic Programming. In 2009 Testing: Academic and Industrial Conference -Practice and Research Techniques. 21--29. https:\/\/doi.org\/10.1109\/TAICPART.2009.18"},{"key":"e_1_3_2_1_25_1","volume-title":"3rd International Symposium on Empirical Software Engineering and Measurement. 291--301","author":"Mockus A.","year":"2009","unstructured":"A. Mockus , N. Nagappan , and T. T. Dinh-Trong . 2009. Test coverage and post-verification defects: A multiple case study . In 3rd International Symposium on Empirical Software Engineering and Measurement. 291--301 . https:\/\/doi.org\/10.1109\/ESEM. 2009 .5315981 10.1109\/ESEM.2009.5315981 A. Mockus, N. Nagappan, and T. T. Dinh-Trong. 2009. Test coverage and post-verification defects: A multiple case study. In 3rd International Symposium on Empirical Software Engineering and Measurement. 291--301. https:\/\/doi.org\/10.1109\/ESEM.2009.5315981"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2001420.2001461"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/CSMR-WCRE.2014.6747166"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2931037.2931040"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.103"},{"key":"e_1_3_2_1_30_1","volume-title":"Yves Le Traon, and Mark Harman","author":"Papadakis Mike","year":"2019","unstructured":"Mike Papadakis , Marinos Kintis , Jie Zhang , Yue Jia , Yves Le Traon, and Mark Harman . 2019 . Mutation Testing Advances: An Analysis and Survey. In Advances in Computers, Atif M. Memon (Ed.). Advances in Computers, Vol. 112 . Elsevier , 275--378. https:\/\/doi.org\/10.1016\/bs.adcom.2018.03.015 10.1016\/bs.adcom.2018.03.015 Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Mutation Testing Advances: An Analysis and Survey. In Advances in Computers, Atif M. Memon (Ed.). Advances in Computers, Vol. 112. Elsevier, 275--378. https:\/\/doi.org\/10.1016\/bs.adcom.2018.03.015"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3273934.3273939"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183519.3183521"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW.2018.00027"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2005.74"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2018.06.024"},{"key":"e_1_3_2_1_36_1","volume-title":"An Empirical Evaluation of the MuJava Mutation Operators. In Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007","author":"Smith B. H.","year":"2007","unstructured":"B. H. Smith and L. Williams . 2007 . An Empirical Evaluation of the MuJava Mutation Operators. In Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007 ). 193--202. https:\/\/doi.org\/10.1109\/TAIC.PART. 2007 .12 10.1109\/TAIC.PART.2007.12 B. H. Smith and L. Williams. 2007. An Empirical Evaluation of the MuJava Mutation Operators. In Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007). 193--202. https:\/\/doi.org\/10.1109\/TAIC.PART.2007.12"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2970276.2970340"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2015.37"},{"key":"e_1_3_2_1_39_1","volume-title":"Fundamentals of Data Visualization: A primer on making informative and compelling figures","author":"Wilke Claus O.","unstructured":"Claus O. Wilke . 2018. Fundamentals of Data Visualization: A primer on making informative and compelling figures . O'Reilly Media . Claus O. Wilke. 2018. Fundamentals of Data Visualization: A primer on making informative and compelling figures. O'Reilly Media."},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568265"},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2014.06.009"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2786805.2786858"}],"event":{"name":"EASE '20: Evaluation and Assessment in Software Engineering","location":"Trondheim Norway","acronym":"EASE '20","sponsor":["NTNU Norwegian University of Science and Technology"]},"container-title":["Proceedings of the Evaluation and Assessment in Software Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3383219.3383236","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3383219.3383236","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:02:01Z","timestamp":1750197721000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3383219.3383236"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,4,15]]},"references-count":43,"alternative-id":["10.1145\/3383219.3383236","10.1145\/3383219"],"URL":"https:\/\/doi.org\/10.1145\/3383219.3383236","relation":{},"subject":[],"published":{"date-parts":[[2020,4,15]]},"assertion":[{"value":"2020-04-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}