{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,30]],"date-time":"2026-01-30T00:23:54Z","timestamp":1769732634272,"version":"3.49.0"},"publisher-location":"New York, NY, USA","reference-count":97,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,11,8]],"date-time":"2020-11-08T00:00:00Z","timestamp":1604793600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100011199","name":"European Research Council","doi-asserted-by":"publisher","award":["741278"],"award-info":[{"award-number":["741278"]}],"id":[{"id":"10.13039\/100011199","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,11,8]]},"DOI":"10.1145\/3368089.3409742","type":"proceedings-article","created":{"date-parts":[[2020,11,8]],"date-time":"2020-11-08T06:03:52Z","timestamp":1604815432000},"page":"1127-1139","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["Cost measures matter for mutation testing study validity"],"prefix":"10.1145","author":[{"given":"Giovani","family":"Guizzo","sequence":"first","affiliation":[{"name":"University College London, UK"}]},{"given":"Federica","family":"Sarro","sequence":"additional","affiliation":[{"name":"University College London, UK"}]},{"given":"Mark","family":"Harman","sequence":"additional","affiliation":[{"name":"University College London, UK"}]}],"member":"320","published-online":{"date-parts":[[2020,11,8]]},"reference":[{"key":"e_1_3_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/VACE.2017.3"},{"key":"e_1_3_2_2_2_1","volume-title":"Establishing Theoretical Minimal Sets of Mutants. In 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation. 21-30","author":"Ammann P.","unstructured":"P. Ammann , M. E. Delamaro , and J. Ofutt . 2014 . Establishing Theoretical Minimal Sets of Mutants. In 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation. 21-30 . P. Ammann, M. E. Delamaro, and J. Ofutt. 2014. Establishing Theoretical Minimal Sets of Mutants. In 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation. 21-30."},{"key":"e_1_3_2_2_3_1","doi-asserted-by":"crossref","unstructured":"A. Arcuri and L. Briand. 2014. A Hitchhiker's guide to statistical tests for assessing randomized algorithms in software engineering. Software Testing Verification and Reliability 24 3 ( 2014 ) 219-250.  A. Arcuri and L. Briand. 2014. A Hitchhiker's guide to statistical tests for assessing randomized algorithms in software engineering. Software Testing Verification and Reliability 24 3 ( 2014 ) 219-250.","DOI":"10.1002\/stvr.1486"},{"key":"e_1_3_2_2_4_1","doi-asserted-by":"crossref","unstructured":"E. F. Barbosa J. C. Maldonado and A. M. R. Vincenzi. 2001. Toward the determination of suficient mutant operators for C. Software Testing Verification and Reliability 11 2 ( 2001 ) 113-136.  E. F. Barbosa J. C. Maldonado and A. M. R. Vincenzi. 2001. Toward the determination of suficient mutant operators for C. Software Testing Verification and Reliability 11 2 ( 2001 ) 113-136.","DOI":"10.1002\/stvr.226"},{"key":"e_1_3_2_2_5_1","first-page":"41","volume-title":"Proceedings of the 8th International Conference on Dependability and Complex Systems (DepCoS-RELCOMEX)","author":"Bluemke I.","unstructured":"I. Bluemke and K. Kulesza . 2013. Reduction of Computational Cost in Mutation Testing by Sampling Mutants . In Proceedings of the 8th International Conference on Dependability and Complex Systems (DepCoS-RELCOMEX) . Brun\u00f3w, Poland , 41 - 51 . I. Bluemke and K. Kulesza. 2013. Reduction of Computational Cost in Mutation Testing by Sampling Mutants. In Proceedings of the 8th International Conference on Dependability and Complex Systems (DepCoS-RELCOMEX). Brun\u00f3w, Poland, 41-51."},{"key":"e_1_3_2_2_6_1","first-page":"297","volume-title":"Proceedings of the 9th International Conference on Software Engineering and Applications (ICSOFT-EA)","author":"Bluemke I.","unstructured":"I. Bluemke and K. Kulesza . 2014. Reduction in Mutation Testing of Java Classes . In Proceedings of the 9th International Conference on Software Engineering and Applications (ICSOFT-EA) . Vienna, Austria , 297 - 304 . I. Bluemke and K. Kulesza. 2014. Reduction in Mutation Testing of Java Classes. In Proceedings of the 9th International Conference on Software Engineering and Applications (ICSOFT-EA). Vienna, Austria, 297-304."},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-07013-1_9"},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF00625279"},{"key":"e_1_3_2_2_9_1","article-title":"High-Performance Mutation Testing","volume":"20","author":"Choi B.","year":"1993","unstructured":"B. Choi and A. P. Mathur . 1993 . High-Performance Mutation Testing . Journal of Systems and Software 20 , 2 ( 1993 ), 135-152. B. Choi and A. P. Mathur. 1993. High-Performance Mutation Testing. Journal of Systems and Software 20, 2 ( 1993 ), 135-152.","journal-title":"Journal of Systems and Software"},{"key":"e_1_3_2_2_10_1","volume-title":"International Symposium on Software Testing and Analysis. ACM, 449-452","author":"Coles H.","unstructured":"H. Coles , T. Laurent , C. Henard , M. Papadakis , and A. Ventresque . 2016. PIT: A Practical Mutation Testing Tool for Java (Demo) . In International Symposium on Software Testing and Analysis. ACM, 449-452 . H. Coles, T. Laurent, C. Henard, M. Papadakis, and A. Ventresque. 2016. PIT: A Practical Mutation Testing Tool for Java (Demo). In International Symposium on Software Testing and Analysis. ACM, 449-452."},{"key":"e_1_3_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/CEC.2013.6557654"},{"key":"e_1_3_2_2_12_1","volume-title":"Experimental Evaluation of SDL and One-Op Mutation for C. In International Conference on Software Testing, Verification and Validation. 203-212","author":"Delamaro M. E.","unstructured":"M. E. Delamaro , L. Deng , V. H. S. Durelli , N. Li , and J. Ofutt . 2014 . Experimental Evaluation of SDL and One-Op Mutation for C. In International Conference on Software Testing, Verification and Validation. 203-212 . M. E. Delamaro, L. Deng, V. H. S. Durelli, N. Li, and J. Ofutt. 2014. Experimental Evaluation of SDL and One-Op Mutation for C. In International Conference on Software Testing, Verification and Validation. 203-212."},{"key":"e_1_3_2_2_13_1","volume-title":"Proceedings of the 28th Brazilian Symposium on Software Engineering (SBES). 81-90","author":"Delamaro M. E.","unstructured":"M. E. Delamaro , L. Deng , N. Li , V. H. S. Durelli , and A. J. Ofutt . 2014. Growing a Reduced Set of Mutation Operators . In Proceedings of the 28th Brazilian Symposium on Software Engineering (SBES). 81-90 . M. E. Delamaro, L. Deng, N. Li, V. H. S. Durelli, and A. J. Ofutt. 2014. Growing a Reduced Set of Mutation Operators. In Proceedings of the 28th Brazilian Symposium on Software Engineering (SBES). 81-90."},{"key":"e_1_3_2_2_14_1","volume-title":"Proceedings of the 7th International Conference on Software Testing, Verification and Validation (ICST). 11-20","author":"Delamaro M. E.","unstructured":"M. E. Delamaro , A. J. Ofutt , and P. Ammann . 2014. Designing Deletion Mutation Operators . In Proceedings of the 7th International Conference on Software Testing, Verification and Validation (ICST). 11-20 . M. E. Delamaro, A. J. Ofutt, and P. Ammann. 2014. Designing Deletion Mutation Operators. In Proceedings of the 7th International Conference on Software Testing, Verification and Validation (ICST). 11-20."},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2016.07.002"},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1630"},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/CMPSAC.1991.170202"},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2013.20"},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-00945-2_11"},{"key":"e_1_3_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-39639-2_9"},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW.2014.24"},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-18473-9_23"},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-30561-0_5"},{"key":"e_1_3_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.15439\/2017F375"},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2011.03.008"},{"key":"e_1_3_2_2_26_1","doi-asserted-by":"crossref","unstructured":"A. E. Eiben and J. E. Smith. 2003. Introduction to evolutionary computing. Springer Science & Business Media.  A. E. Eiben and J. E. Smith. 2003. Introduction to evolutionary computing. Springer Science & Business Media.","DOI":"10.1007\/978-3-662-05094-1"},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"crossref","unstructured":"A. Finkelstein M. Harman Y. Jia W. Martin F. Sarro and Y. Zhang. 2017. Investigating the relationship between price rating and popularity in the Blackberry World App Store. Information and Software Technology 87 ( 2017 ) 119-139.  A. Finkelstein M. Harman Y. Jia W. Martin F. Sarro and Y. Zhang. 2017. Investigating the relationship between price rating and popularity in the Blackberry World App Store. Information and Software Technology 87 ( 2017 ) 119-139.","DOI":"10.1016\/j.infsof.2017.03.002"},{"key":"e_1_3_2_2_28_1","volume-title":"Evolutionary Generation of Whole Test Suites. In International Conference On Quality Software (QSIC). 31-40","author":"Fraser G.","unstructured":"G. Fraser and A. Arcuri . 2011 . Evolutionary Generation of Whole Test Suites. In International Conference On Quality Software (QSIC). 31-40 . G. Fraser and A. Arcuri. 2011. Evolutionary Generation of Whole Test Suites. In International Conference On Quality Software (QSIC). 31-40."},{"key":"e_1_3_2_2_29_1","first-page":"224","volume-title":"Proceedings of the 22nd International Symposium on Software Testing and Analysis (ISSTA)","author":"Gligoric M.","unstructured":"M. Gligoric , L. Zhang , C. Pereira , and G. Pokam . 2013. Selective Mutation Testing for Concurrent Code . In Proceedings of the 22nd International Symposium on Software Testing and Analysis (ISSTA) . Lugano, Switzerland , 224 - 234 . M. Gligoric, L. Zhang, C. Pereira, and G. Pokam. 2013. Selective Mutation Testing for Concurrent Code. In Proceedings of the 22nd International Symposium on Software Testing and Analysis (ISSTA). Lugano, Switzerland, 224-234."},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"crossref","unstructured":"D. Gong G. Zhang X. Yao and F. Meng. 2017. Mutant reduction based on dominance relation for weak mutation testing. Information and Software Technology 81 ( 2017 ) 82-96.  D. Gong G. Zhang X. Yao and F. Meng. 2017. Mutant reduction based on dominance relation for weak mutation testing. Information and Software Technology 81 ( 2017 ) 82-96.","DOI":"10.1016\/j.infsof.2016.05.001"},{"key":"e_1_3_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/TR.2017.2705662"},{"key":"e_1_3_2_2_32_1","volume-title":"2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE). 216-227","author":"Gopinath R.","unstructured":"R. Gopinath , A. Alipour , I. Ahmed , C. Jensen , and A. Groce . 2015. How hard does mutation analysis have to be, anyway? . In 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE). 216-227 . R. Gopinath, A. Alipour, I. Ahmed, C. Jensen, and A. Groce. 2015. How hard does mutation analysis have to be, anyway?. In 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE). 216-227."},{"key":"e_1_3_2_2_33_1","volume-title":"On The Limits of Mutation Reduction Strategies. 2016 IEEE\/ACM 38th International Conference on Software Engineering (ICSE) ( 2016 ), 511-522","author":"Gopinath R.","unstructured":"R. Gopinath , M. A. Alipour , I. Ahmed , C. Jensen , and A. Groce . 2016 . On The Limits of Mutation Reduction Strategies. 2016 IEEE\/ACM 38th International Conference on Software Engineering (ICSE) ( 2016 ), 511-522 . R. Gopinath, M. A. Alipour, I. Ahmed, C. Jensen, and A. Groce. 2016. On The Limits of Mutation Reduction Strategies. 2016 IEEE\/ACM 38th International Conference on Software Engineering (ICSE) ( 2016 ), 511-522."},{"key":"e_1_3_2_2_34_1","volume-title":"Sentinel: A Hyper-Heuristic for the Generation of Mutant Reduction Strategies. Transactions on Software Engineering ( 2020 ).","author":"Guizzo G.","year":"2020","unstructured":"G. Guizzo , F. Sarro , J. Krinke , and S. R. Vergilio . 2020 . Sentinel: A Hyper-Heuristic for the Generation of Mutant Reduction Strategies. Transactions on Software Engineering ( 2020 ). G. Guizzo, F. Sarro, J. Krinke, and S. R. Vergilio. 2020. Sentinel: A Hyper-Heuristic for the Generation of Mutant Reduction Strategies. Transactions on Software Engineering ( 2020 )."},{"key":"e_1_3_2_2_35_1","volume-title":"International Conference on Software Testing, Verification, and Validation Workshops. 80-89","author":"Harman M.","unstructured":"M. Harman , Y. Jia , and W. B. Langdon . 2010. A Manifesto for Higher Order Mutation Testing . In International Conference on Software Testing, Verification, and Validation Workshops. 80-89 . M. Harman, Y. Jia, and W. B. Langdon. 2010. A Manifesto for Higher Order Mutation Testing. In International Conference on Software Testing, Verification, and Validation Workshops. 80-89."},{"key":"e_1_3_2_2_36_1","volume-title":"International Conference on Automated Software Engineering. 397-408","author":"Harman M.","unstructured":"M. Harman , Y. Jia , P. R. Mateo , and M. Polo . 2014. Angels and monsters: An empirical investigation of potential test efectiveness and eficiency improvement from strongly subsuming higher order mutation . In International Conference on Automated Software Engineering. 397-408 . M. Harman, Y. Jia, P. R. Mateo, and M. Polo. 2014. Angels and monsters: An empirical investigation of potential test efectiveness and eficiency improvement from strongly subsuming higher order mutation. In International Conference on Automated Software Engineering. 397-408."},{"key":"e_1_3_2_2_37_1","article-title":"Weak Mutation Testing and Completeness of Test Sets","volume":"8","author":"Howden W. E.","year":"1982","unstructured":"W. E. Howden . 1982 . Weak Mutation Testing and Completeness of Test Sets . IEEE Transactions on Software Engineering 8 , 4 ( 1982 ), 371-379. W. E. Howden. 1982. Weak Mutation Testing and Completeness of Test Sets. IEEE Transactions on Software Engineering 8, 4 ( 1982 ), 371-379.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW.2017.29"},{"key":"e_1_3_2_2_39_1","first-page":"639","volume-title":"Proceedings of the 9th Joint Meeting on Foundations of Software Engineering: New Ideas Track (ESEC\/FSE)","author":"Inozemtseva L.","unstructured":"L. Inozemtseva , H. Hemmati , and R. Holmes . 2013. Using Fault History to Improve Mutation Reduction . In Proceedings of the 9th Joint Meeting on Foundations of Software Engineering: New Ideas Track (ESEC\/FSE) . Saint Petersburg, Russia , 639 - 642 . L. Inozemtseva, H. Hemmati, and R. Holmes. 2013. Using Fault History to Improve Mutation Reduction. In Proceedings of the 9th Joint Meeting on Foundations of Software Engineering: New Ideas Track (ESEC\/FSE). Saint Petersburg, Russia, 639-642."},{"key":"e_1_3_2_2_40_1","first-page":"422","volume-title":"International Conference on Software Engineering and Knowledge Engineering","volume":"9","author":"Ji C.","unstructured":"C. Ji , Z. Chen , B. Xu , and Z. Zhao . 2009. A Novel Method of Mutation Clustering Based on Domain Analysis . In International Conference on Software Engineering and Knowledge Engineering , Vol. 9 . 422 - 425 . C. Ji, Z. Chen, B. Xu, and Z. Zhao. 2009. A Novel Method of Mutation Clustering Based on Domain Analysis. In International Conference on Software Engineering and Knowledge Engineering, Vol. 9. 422-425."},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2010.62"},{"key":"e_1_3_2_2_42_1","first-page":"654","article-title":"Are Mutants a Valid Substitute for Real Faults in Software Testing?","author":"Just R.","year":"2014","unstructured":"R. Just , D. Jalali , L. Inozemtseva , M. D. Ernst , R. Holmes , and G. Fraser . 2014 . Are Mutants a Valid Substitute for Real Faults in Software Testing? . In Proc. of FSE. 654 - 665 . R. Just, D. Jalali, L. Inozemtseva, M. D. Ernst, R. Holmes, and G. Fraser. 2014. Are Mutants a Valid Substitute for Real Faults in Software Testing?. In Proc. of FSE. 654-665.","journal-title":"Proc. of FSE."},{"key":"e_1_3_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3092703.3092732"},{"key":"e_1_3_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1561"},{"key":"e_1_3_2_2_45_1","first-page":"386","volume-title":"Proceedings of the 2nd International Conference on Software Testing, Verification and Validation (ICST)","author":"Kaminski G.","unstructured":"G. Kaminski and P. Ammann . 2009. Using a Fault Hierarchy to Improve the Eficiency of DNF Logic Mutation Testing . In Proceedings of the 2nd International Conference on Software Testing, Verification and Validation (ICST) . Denver, CO, USA , 386 - 395 . G. Kaminski and P. Ammann. 2009. Using a Fault Hierarchy to Improve the Eficiency of DNF Logic Mutation Testing. In Proceedings of the 2nd International Conference on Software Testing, Verification and Validation (ICST). Denver, CO, USA, 386-395."},{"key":"e_1_3_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2012.08.024"},{"key":"e_1_3_2_2_47_1","doi-asserted-by":"crossref","unstructured":"G. Kaminski U. Praphamontripong P. Ammann and A. J. Ofutt. 2011. A Logic Mutation Approach to Selective Mutation for Programs and Queries. Information and Software Technology 53 10 ( 2011 ) 1137-1152.  G. Kaminski U. Praphamontripong P. Ammann and A. J. Ofutt. 2011. A Logic Mutation Approach to Selective Mutation for Programs and Queries. Information and Software Technology 53 10 ( 2011 ) 1137-1152.","DOI":"10.1016\/j.infsof.2011.03.009"},{"key":"e_1_3_2_2_48_1","doi-asserted-by":"crossref","unstructured":"K. N. King and A. J. Ofutt. 1991. A Fortran Language System for Mutation-based Software Testing. Software-Practice and Experience 21 7 ( 1991 ) 685-718.  K. N. King and A. J. Ofutt. 1991. A Fortran Language System for Mutation-based Software Testing. Software-Practice and Experience 21 7 ( 1991 ) 685-718.","DOI":"10.1002\/spe.4380210704"},{"key":"e_1_3_2_2_49_1","first-page":"300","volume-title":"Proceedings of the 17th Asia-Pacific Software Engineering Conference (APSEC)","author":"Kintis M.","unstructured":"M. Kintis , M. Papadakis , and N. Malevris . 2010. Evaluating Mutation Testing Alternatives: A Collateral Experiment . In Proceedings of the 17th Asia-Pacific Software Engineering Conference (APSEC) . Sydney, Australia , 300 - 309 . M. Kintis, M. Papadakis, and N. Malevris. 2010. Evaluating Mutation Testing Alternatives: A Collateral Experiment. In Proceedings of the 17th Asia-Pacific Software Engineering Conference (APSEC). Sydney, Australia, 300-309."},{"key":"e_1_3_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/2950290.2950322"},{"key":"e_1_3_2_2_51_1","first-page":"21","volume-title":"Proceedings of the 8th Brazilian Workshop on Systematic and Automated Software Testing (SAST)","author":"Lacerda J. T. S.","unstructured":"J. T. S. Lacerda and F. C. Ferrari . 2014. Towards the Establishment of a Suficient Set of Mutation Operators for AspectJ Programs . In Proceedings of the 8th Brazilian Workshop on Systematic and Automated Software Testing (SAST) . Maceio, AL, Brazil , 21 - 30 . J. T. S. Lacerda and F. C. Ferrari. 2014. Towards the Establishment of a Suficient Set of Mutation Operators for AspectJ Programs. In Proceedings of the 8th Brazilian Workshop on Systematic and Automated Software Testing (SAST). Maceio, AL, Brazil, 21-30."},{"key":"e_1_3_2_2_52_1","volume-title":"Assessing and Improving the Mutation Testing Practice of PIT. In 2017 IEEE International Conference on Software Testing, Verification and Validation. 430-435","author":"Laurent T.","unstructured":"T. Laurent , M. Papadakis , M. Kintis , C. Henard , Y. L. Traon , and A. Ventresque . 2017 . Assessing and Improving the Mutation Testing Practice of PIT. In 2017 IEEE International Conference on Software Testing, Verification and Validation. 430-435 . T. Laurent, M. Papadakis, M. Kintis, C. Henard, Y. L. Traon, and A. Ventresque. 2017. Assessing and Improving the Mutation Testing Practice of PIT. In 2017 IEEE International Conference on Software Testing, Verification and Validation. 430-435."},{"key":"e_1_3_2_2_53_1","volume-title":"Evaluating Diferent Strategies for Reduction of Mutation Testing Costs. In Simp\u00f3sio Brasileiro de Teste de Software Sistem\u00e1tico e Automatizado.","author":"Lima J. A. P.","unstructured":"J. A. P. Lima , G. Guizzo , S. R. Vergilio , A. P. C. Silva , H. L. J. Filho , and H. V. Ehrenfried . 2016 . Evaluating Diferent Strategies for Reduction of Mutation Testing Costs. In Simp\u00f3sio Brasileiro de Teste de Software Sistem\u00e1tico e Automatizado. J. A. P. Lima, G. Guizzo, S. R. Vergilio, A. P. C. Silva, H. L. J. Filho, and H. V. Ehrenfried. 2016. Evaluating Diferent Strategies for Reduction of Mutation Testing Costs. In Simp\u00f3sio Brasileiro de Teste de Software Sistem\u00e1tico e Automatizado."},{"key":"e_1_3_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.4218\/etrij.09.0108.0356"},{"key":"e_1_3_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2013.44"},{"key":"e_1_3_2_2_56_1","article-title":"Validating Second-Order Mutation at System Level","volume":"39","author":"Mateo P. R.","year":"2013","unstructured":"P. R. Mateo , M. P. Usaola , and J. L. F. Alem\u00e1n . 2013 . Validating Second-Order Mutation at System Level . IEEE Transactions on Software Engineering 39 , 4 ( 2013 ), 570-587. P. R. Mateo, M. P. Usaola, and J. L. F. Alem\u00e1n. 2013. Validating Second-Order Mutation at System Level. IEEE Transactions on Software Engineering 39, 4 ( 2013 ), 570-587.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_2_2_57_1","volume-title":"Computer Software and Applications Conference. 604-605","author":"Mathur A. P.","year":"1991","unstructured":"A. P. Mathur . 1991 . Performance, efectiveness, and reliability issues in software testing . In Computer Software and Applications Conference. 604-605 . A. P. Mathur. 1991. Performance, efectiveness, and reliability issues in software testing. In Computer Software and Applications Conference. 604-605."},{"key":"e_1_3_2_2_58_1","doi-asserted-by":"crossref","unstructured":"A. P. Mathur and W. E. Wong. 1994. An empirical comparison of data flow and mutation-based test adequacy criteria. Software Testing Verification and Reliability 4 1 ( 1994 ) 9-31.  A. P. Mathur and W. E. Wong. 1994. An empirical comparison of data flow and mutation-based test adequacy criteria. Software Testing Verification and Reliability 4 1 ( 1994 ) 9-31.","DOI":"10.1002\/stvr.4370040104"},{"key":"e_1_3_2_2_59_1","doi-asserted-by":"crossref","unstructured":"E. S. Mresa and L. Bottaci. 1999. Eficiency of Mutation Operators and Selective Mutation Strategies: an Empirical Study. Software Testing Verification and Reliability 9 4 ( 1999 ) 205-232.  E. S. Mresa and L. Bottaci. 1999. Eficiency of Mutation Operators and Selective Mutation Strategies: an Empirical Study. Software Testing Verification and Reliability 9 4 ( 1999 ) 205-232.","DOI":"10.1002\/(SICI)1099-1689(199912)9:4<205::AID-STVR186>3.0.CO;2-X"},{"key":"e_1_3_2_2_60_1","volume-title":"Calibrated Mutation Testing. In 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops. 376-381","author":"Nam J.","unstructured":"J. Nam , D. Schuler , and A. Zeller . 2011 . Calibrated Mutation Testing. In 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops. 376-381 . J. Nam, D. Schuler, and A. Zeller. 2011. Calibrated Mutation Testing. In 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops. 376-381."},{"key":"e_1_3_2_2_61_1","volume-title":"International Conference on Software Engineering. 351-360","author":"Namin A. S.","unstructured":"A. S. Namin , J. Andrews , and D. Murdoch . 2008. Suficient mutation operators for measuring test efectiveness . In International Conference on Software Engineering. 351-360 . A. S. Namin, J. Andrews, and D. Murdoch. 2008. Suficient mutation operators for measuring test efectiveness. In International Conference on Software Engineering. 351-360."},{"key":"e_1_3_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.4018\/jncr.2012070102"},{"key":"e_1_3_2_2_63_1","article-title":"An Experimental Determination of Suficient Mutant Operators","volume":"5","author":"Ofutt A. J.","year":"1996","unstructured":"A. J. Ofutt , A. Lee , G. Rothermel , R. H. Untch , and C. Zapf . 1996 . An Experimental Determination of Suficient Mutant Operators . ACM Transactions on Software Engineering and Methodology 5 , 2 ( 1996 ), 99-118. A. J. Ofutt, A. Lee, G. Rothermel, R. H. Untch, and C. Zapf. 1996. An Experimental Determination of Suficient Mutant Operators. ACM Transactions on Software Engineering and Methodology 5, 2 ( 1996 ), 99-118.","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"key":"e_1_3_2_2_64_1","volume-title":"International Conference on Software Engineering. 100-107","author":"Ofutt A. J.","unstructured":"A. J. Ofutt , G. Rothermel , and C. Zapf . 1993. An experimental evaluation of selective mutation . In International Conference on Software Engineering. 100-107 . A. J. Ofutt, G. Rothermel, and C. Zapf. 1993. An experimental evaluation of selective mutation. In International Conference on Software Engineering. 100-107."},{"key":"e_1_3_2_2_65_1","doi-asserted-by":"crossref","unstructured":"A. J. Ofutt and R. H. Untch. 2001. Mutation Testing for the New Century. Springer Chapter Mutation 2000: Uniting the Orthogonal 34-44.  A. J. Ofutt and R. H. Untch. 2001. Mutation Testing for the New Century. Springer Chapter Mutation 2000: Uniting the Orthogonal 34-44.","DOI":"10.1007\/978-1-4757-5939-6_7"},{"key":"e_1_3_2_2_66_1","first-page":"1","volume-title":"Proceedings of the 23th International Symposium on Software Reliability Engineering (ISSRE)","author":"Omar E.","unstructured":"E. Omar and S. Ghosh . 2012. An Exploratory Study of Higher Order Mutation Testing in Aspect-Oriented Programming . In Proceedings of the 23th International Symposium on Software Reliability Engineering (ISSRE) . Dallas, TX, USA , 1 - 10 . E. Omar and S. Ghosh. 2012. An Exploratory Study of Higher Order Mutation Testing in Aspect-Oriented Programming. In Proceedings of the 23th International Symposium on Software Reliability Engineering (ISSRE). Dallas, TX, USA, 1-10."},{"key":"e_1_3_2_2_67_1","unstructured":"M. Papadakis M. Kintis J. Zhang Y. Jia Y. Le Traon and M. Harman. 2017. Mutation Testing Advances: An Analysis and Survey. Advances in Computers ( 2017 ).  M. Papadakis M. Kintis J. Zhang Y. Jia Y. Le Traon and M. Harman. 2017. Mutation Testing Advances: An Analysis and Survey. Advances in Computers ( 2017 )."},{"key":"e_1_3_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/2554850.2554978"},{"key":"e_1_3_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1509"},{"key":"e_1_3_2_2_70_1","volume-title":"An Empirical Evaluation of the First and Second Order Mutation Testing Strategies. In International Conference on Software Testing, Verification and Validation Workshops. 90-99","author":"Papadakis M.","unstructured":"M. Papadakis and N. Malevris . 2010 . An Empirical Evaluation of the First and Second Order Mutation Testing Strategies. In International Conference on Software Testing, Verification and Validation Workshops. 90-99 . M. Papadakis and N. Malevris. 2010. An Empirical Evaluation of the First and Second Order Mutation Testing Strategies. In International Conference on Software Testing, Verification and Validation Workshops. 90-99."},{"key":"e_1_3_2_2_71_1","volume-title":"2016 IEEE International Conference on Software Quality, Reliability and Security (QRS). IEEE, 365-373","author":"Parsai A.","unstructured":"A. Parsai , A. Murgia , and S. Demeyer . 2016. A model to estimate first-order mutation coverage from higher-order mutation coverage . In 2016 IEEE International Conference on Software Quality, Reliability and Security (QRS). IEEE, 365-373 . A. Parsai, A. Murgia, and S. Demeyer. 2016. A model to estimate first-order mutation coverage from higher-order mutation coverage. In 2016 IEEE International Conference on Software Quality, Reliability and Security (QRS). IEEE, 365-373."},{"key":"e_1_3_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/2915970.2915992"},{"key":"e_1_3_2_2_73_1","volume-title":"2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops. IEEE, 186-195","author":"Patrick M.","unstructured":"M. Patrick , R. Alexander , M. Oriol , and J. A. Clark . 2014. Probability-based semantic interpretation of mutants . In 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops. IEEE, 186-195 . M. Patrick, R. Alexander, M. Oriol, and J. A. Clark. 2014. Probability-based semantic interpretation of mutants. In 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops. IEEE, 186-195."},{"key":"e_1_3_2_2_74_1","volume-title":"Proceedings of the 5th International Conference on Software Testing, Verification and Validation (ICST). Montreal, QC, Canada, 711-719","author":"Patrick M.","unstructured":"M. Patrick , M. Oriol , and J. A. Clark . 2012. MESSI: Mutant Evaluation by Static Semantic Interpretation . In Proceedings of the 5th International Conference on Software Testing, Verification and Validation (ICST). Montreal, QC, Canada, 711-719 . M. Patrick, M. Oriol, and J. A. Clark. 2012. MESSI: Mutant Evaluation by Static Semantic Interpretation. In Proceedings of the 5th International Conference on Software Testing, Verification and Validation (ICST). Montreal, QC, Canada, 711-719."},{"key":"e_1_3_2_2_75_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2019.07.100"},{"key":"e_1_3_2_2_76_1","doi-asserted-by":"crossref","unstructured":"M. Polo M. Piattini and I. Garc\u00eda-Rodr\u00edguez. 2009. Decreasing the cost of mutation testing with second-order mutants. Software Testing Verification and Reliability 19 2 ( 2009 ) 111-131.  M. Polo M. Piattini and I. Garc\u00eda-Rodr\u00edguez. 2009. Decreasing the cost of mutation testing with second-order mutants. Software Testing Verification and Reliability 19 2 ( 2009 ) 111-131.","DOI":"10.1002\/stvr.392"},{"key":"e_1_3_2_2_77_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW.2017.30"},{"key":"e_1_3_2_2_78_1","doi-asserted-by":"publisher","DOI":"10.1109\/ELINFOCOM.2016.7562970"},{"key":"e_1_3_2_2_79_1","first-page":"131","volume-title":"Proceedings of the 19th International Conference on Software Product Line (SPLC)","author":"Reuling D.","unstructured":"D. Reuling , J. B\u00fcrdek , S. Rot\u00e4rmel , M. Lochau , and U. Kelter . 2015. Fault-based Product-line Testing: Efective Sample Generation Based on Feature-Diagram Mutation . In Proceedings of the 19th International Conference on Software Product Line (SPLC) . Nashville, TN, USA , 131 - 140 . D. Reuling, J. B\u00fcrdek, S. Rot\u00e4rmel, M. Lochau, and U. Kelter. 2015. Fault-based Product-line Testing: Efective Sample Generation Based on Feature-Diagram Mutation. In Proceedings of the 19th International Conference on Software Product Line (SPLC). Nashville, TN, USA, 131-140."},{"key":"e_1_3_2_2_80_1","doi-asserted-by":"publisher","DOI":"10.1109\/MUTATION.2006.7"},{"key":"e_1_3_2_2_81_1","doi-asserted-by":"crossref","unstructured":"C. Spearman. 1904. The proof and measurement of association between two things. American journal of Psychology 15 1 ( 1904 ) 72-101.  C. Spearman. 1904. The proof and measurement of association between two things. American journal of Psychology 15 1 ( 1904 ) 72-101.","DOI":"10.2307\/1412159"},{"key":"e_1_3_2_2_82_1","first-page":"378","volume-title":"Proceedings of the IEEE 21th International Symposium on Software Reliability Engineering (ISSRE)","author":"Sridharan M.","unstructured":"M. Sridharan and A. Siami-Namin . 2010. Prioritizing Mutation Operators based on Importance Sampling . In Proceedings of the IEEE 21th International Symposium on Software Reliability Engineering (ISSRE) . San Jose, CA, USA , 378 - 387 . M. Sridharan and A. Siami-Namin. 2010. Prioritizing Mutation Operators based on Importance Sampling. In Proceedings of the IEEE 21th International Symposium on Software Reliability Engineering (ISSRE). San Jose, CA, USA, 378-387."},{"key":"e_1_3_2_2_83_1","volume-title":"Proceedings of the 32th International Conference on Software Engineering (ICSE). Cape Town, South Africa, 425-434","author":"Steimann F.","unstructured":"F. Steimann and A. Thies . 2010. From Behaviour Preservation to Behaviour Modification: Constraint-Based Mutant Generation . In Proceedings of the 32th International Conference on Software Engineering (ICSE). Cape Town, South Africa, 425-434 . F. Steimann and A. Thies. 2010. From Behaviour Preservation to Behaviour Modification: Constraint-Based Mutant Generation. In Proceedings of the 32th International Conference on Software Engineering (ICSE). Cape Town, South Africa, 425-434."},{"key":"e_1_3_2_2_84_1","doi-asserted-by":"publisher","DOI":"10.1093\/comjnl\/bxw076"},{"key":"e_1_3_2_2_85_1","doi-asserted-by":"crossref","unstructured":"C. Sun F. Xue H. Liu and X. Zhang. 2017. A Path-aware Approach to Mutant Reduction in Mutation Testing. Information and Software Technology 81 ( 2017 ) 65-81.  C. Sun F. Xue H. Liu and X. Zhang. 2017. A Path-aware Approach to Mutant Reduction in Mutation Testing. Information and Software Technology 81 ( 2017 ) 65-81.","DOI":"10.1016\/j.infsof.2016.02.006"},{"key":"e_1_3_2_2_86_1","doi-asserted-by":"crossref","unstructured":"J. Tuya M. J. Su\u00e1rez-Cabal and C. de la Riva. 2007. Mutating database queries. Information and Software Technology 49 4 ( 2007 ) 398-417.  J. Tuya M. J. Su\u00e1rez-Cabal and C. de la Riva. 2007. Mutating database queries. Information and Software Technology 49 4 ( 2007 ) 398-417.","DOI":"10.1016\/j.infsof.2006.06.009"},{"key":"e_1_3_2_2_87_1","doi-asserted-by":"publisher","DOI":"10.1145\/1566445.1566540"},{"key":"e_1_3_2_2_88_1","doi-asserted-by":"crossref","unstructured":"A. M. R. Vincenzi J. C. Maldonado E. F. Barbosa and M. E. Delamaro. 2001. Unit and Integration Testing Strategies for C Programs Using Mutation. Software Testing Verification and Reliability 11 4 ( 2001 ) 249-268.  A. M. R. Vincenzi J. C. Maldonado E. F. Barbosa and M. E. Delamaro. 2001. Unit and Integration Testing Strategies for C Programs Using Mutation. Software Testing Verification and Reliability 11 4 ( 2001 ) 249-268.","DOI":"10.1002\/stvr.242"},{"key":"e_1_3_2_2_89_1","doi-asserted-by":"publisher","DOI":"10.1016\/0164-1212(94)00098-0"},{"key":"e_1_3_2_2_90_1","first-page":"258","volume-title":"All-uses: An Empirical Evaluation of Cost, Strength and Efectiveness","author":"Wong W. E.","year":"1995","unstructured":"W. E. Wong , A. P. Mathur , and J. C. Maldonado . 1995 . Software Quality and Productivity: Theory , practice, education and training. Springer , Chapter Mutation versus All-uses: An Empirical Evaluation of Cost, Strength and Efectiveness , 258 - 265 . W. E. Wong, A. P. Mathur, and J. C. Maldonado. 1995. Software Quality and Productivity: Theory, practice, education and training. Springer, Chapter Mutation versus All-uses: An Empirical Evaluation of Cost, Strength and Efectiveness, 258-265."},{"key":"e_1_3_2_2_91_1","volume-title":"Workshop on Software Testing, Verification, and Analysis. 152-158","author":"Woodward M. R.","unstructured":"M. R. Woodward and K. Halewood . 1988. From weak to strong, dead or alive? an analysis of some mutation testing issues . In Workshop on Software Testing, Verification, and Analysis. 152-158 . M. R. Woodward and K. Halewood. 1988. From weak to strong, dead or alive? an analysis of some mutation testing issues. In Workshop on Software Testing, Verification, and Analysis. 152-158."},{"key":"e_1_3_2_2_92_1","volume-title":"Predictive Mutation Testing. In International Symposium on Software Testing and Analysis. 342-353","author":"Zhang J.","unstructured":"J. Zhang , Z. Wang , L. Zhang , D. Hao , L. Zang , S. Cheng , and L. Zhang . 2016 . Predictive Mutation Testing. In International Symposium on Software Testing and Analysis. 342-353 . J. Zhang, Z. Wang, L. Zhang, D. Hao, L. Zang, S. Cheng, and L. Zhang. 2016. Predictive Mutation Testing. In International Symposium on Software Testing and Analysis. 342-353."},{"key":"e_1_3_2_2_93_1","volume-title":"2014 IEEE 25th International Symposium on Software Reliability Engineering. IEEE, 277-287","author":"Zhang J.","unstructured":"J. Zhang , M. Zhu , D. Hao , and L. Zhang . 2014. An empirical study on the scalability of selective mutation testing . In 2014 IEEE 25th International Symposium on Software Reliability Engineering. IEEE, 277-287 . J. Zhang, M. Zhu, D. Hao, and L. Zhang. 2014. An empirical study on the scalability of selective mutation testing. In 2014 IEEE 25th International Symposium on Software Reliability Engineering. IEEE, 277-287."},{"key":"e_1_3_2_2_94_1","volume-title":"2013 28th IEEE\/ACM International Conference on Automated Software Engineering (ASE). 92-102","author":"Zhang L.","unstructured":"L. Zhang , M. Gligoric , D. Marinov , and S. Khurshid . 2013. Operator-based and random mutant selection: Better together . In 2013 28th IEEE\/ACM International Conference on Automated Software Engineering (ASE). 92-102 . L. Zhang, M. Gligoric, D. Marinov, and S. Khurshid. 2013. Operator-based and random mutant selection: Better together. In 2013 28th IEEE\/ACM International Conference on Automated Software Engineering (ASE). 92-102."},{"key":"e_1_3_2_2_95_1","first-page":"435","volume-title":"Proc. of the 32Nd ACM\/IEEE International Conference on Software Engineering (ICSE'10)","author":"Zhang L.","unstructured":"L. Zhang , S.-S. Hou , J.-J. Hu , T. Xie , and H. Mei . 2010. Is Operator-based Mutant Selection Superior to Random Mutant Selection? . In Proc. of the 32Nd ACM\/IEEE International Conference on Software Engineering (ICSE'10) . 435 - 444 . L. Zhang, S.-S. Hou, J.-J. Hu, T. Xie, and H. Mei. 2010. Is Operator-based Mutant Selection Superior to Random Mutant Selection?. In Proc. of the 32Nd ACM\/IEEE International Conference on Software Engineering (ICSE'10). 435-444."},{"key":"e_1_3_2_2_96_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW.2017.25"},{"key":"e_1_3_2_2_97_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2018.00035"}],"event":{"name":"ESEC\/FSE '20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering","location":"Virtual Event USA","acronym":"ESEC\/FSE '20","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering"]},"container-title":["Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3368089.3409742","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3368089.3409742","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:01:57Z","timestamp":1750197717000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3368089.3409742"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,11,8]]},"references-count":97,"alternative-id":["10.1145\/3368089.3409742","10.1145\/3368089"],"URL":"https:\/\/doi.org\/10.1145\/3368089.3409742","relation":{},"subject":[],"published":{"date-parts":[[2020,11,8]]},"assertion":[{"value":"2020-11-08","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}