{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,28]],"date-time":"2026-03-28T21:07:30Z","timestamp":1774732050808,"version":"3.50.1"},"reference-count":80,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2019,12,18]],"date-time":"2019-12-18T00:00:00Z","timestamp":1576627200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2019,12,18]],"date-time":"2019-12-18T00:00:00Z","timestamp":1576627200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001866","name":"Fonds National de la Recherche Luxembourg","doi-asserted-by":"publisher","award":["AFR-PhD"],"award-info":[{"award-number":["AFR-PhD"]}],"id":[{"id":"10.13039\/501100001866","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2020,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Mutant selection refers to the problem of choosing, among a large number of mutants, the (few) ones that should be used by the testers. In view of this, we investigate the problem of selecting the fault revealing mutants, i.e., the mutants that are killable and lead to test cases that uncover unknown program faults. We formulate two variants of this problem: the fault revealing mutant selection and the fault revealing mutant prioritization. We argue and show that these problems can be tackled through a set of \u2018static\u2019 program features and propose a machine learning approach, named <jats:italic>FaRM<\/jats:italic>, that learns to select and rank killable and fault revealing mutants. Experimental results involving 1,692 real faults show the practical benefits of our approach in both examined problems. Our results show that <jats:italic>FaRM<\/jats:italic> achieves a good trade-off between application cost and effectiveness (measured in terms of faults revealed). We also show that <jats:italic>FaRM<\/jats:italic> outperforms all the existing mutant selection methods, i.e., the random mutant sampling, the selective mutation and defect prediction (mutating the code areas pointed by defect prediction). In particular, our results show that with respect to mutant selection, our approach reveals 23% to 34% more faults than any of the baseline methods, while, with respect to mutant prioritization, it achieves higher average percentage of revealed faults with a median difference between 4% and 9% (from the random mutant orderings).<\/jats:p>","DOI":"10.1007\/s10664-019-09778-7","type":"journal-article","created":{"date-parts":[[2019,12,18]],"date-time":"2019-12-18T17:02:42Z","timestamp":1576688562000},"page":"434-487","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":55,"title":["Selecting fault revealing mutants"],"prefix":"10.1007","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5295-1831","authenticated-orcid":false,"given":"Thierry","family":"Titcheu Chekam","sequence":"first","affiliation":[]},{"given":"Mike","family":"Papadakis","sequence":"additional","affiliation":[]},{"given":"Tegawend\u00e9 F.","family":"Bissyand\u00e9","sequence":"additional","affiliation":[]},{"given":"Yves","family":"Le Traon","sequence":"additional","affiliation":[]},{"given":"Koushik","family":"Sen","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2019,12,18]]},"reference":[{"key":"9778_CR1","unstructured":"Acree AT (1980) On mutation. phdthesis Georgia Institute of Technology, Atlanta"},{"key":"9778_CR2","doi-asserted-by":"crossref","unstructured":"Ammann P, Offutt J (2008) Introduction to software testing. Cambridge University Press, Cambridge","DOI":"10.1017\/CBO9780511809163"},{"key":"9778_CR3","doi-asserted-by":"publisher","unstructured":"Ammann P, Delamaro ME, Offutt J (2014) Establishing theoretical minimal sets of mutants. In: Seventh IEEE international conference on software testing, verification and validation, ICST 2014, Cleveland, pp 21\u201330. https:\/\/doi.org\/10.1109\/ICST.2014.13","DOI":"10.1109\/ICST.2014.13"},{"issue":"2","key":"9778_CR4","doi-asserted-by":"publisher","first-page":"113","DOI":"10.1002\/stvr.226","volume":"11","author":"EF Barbosa","year":"2001","unstructured":"Barbosa EF, Maldonado JC, Vincenzi AMR (2001) Toward the determination of sufficient mutant operators for c. Softw Test Verif Reliab 11(2):113\u2013136","journal-title":"Softw Test Verif Reliab"},{"key":"9778_CR5","doi-asserted-by":"publisher","unstructured":"B\u00f6hme M, Roychoudhury A (2014) Corebench: studying complexity of regression errors. In: International symposium on software testing and analysis, ISSTA \u201914, San Jose, pp 105\u2013115. https:\/\/doi.org\/10.1145\/2610384.2628058","DOI":"10.1145\/2610384.2628058"},{"issue":"1","key":"9778_CR6","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1023\/A:1010933404324","volume":"45","author":"L Breiman","year":"2001","unstructured":"Breiman L (2001) Random forests. Mach Learn 45(1):5\u201332","journal-title":"Mach Learn"},{"key":"9778_CR7","unstructured":"Budd TA (1980) Mutation analysis of program test data. phdthesis Yale University, New Haven"},{"issue":"1","key":"9778_CR8","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1007\/BF00625279","volume":"18","author":"TA Budd","year":"1982","unstructured":"Budd TA, Angluin D (1982) Two notions of correctness and their relation to testing. Acta Inf 18(1):31\u201345. https:\/\/doi.org\/10.1007\/BF00625279","journal-title":"Acta Inf"},{"key":"9778_CR9","unstructured":"Cadar C, Dunbar D, Engler DR (2008) KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs. In: 8th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2008. Proceedings, San Diego, pp 209\u2013224. http:\/\/www.usenix.org\/events\/osdi08\/tech\/full_papers\/cadar\/cadar.pdf"},{"key":"9778_CR10","doi-asserted-by":"publisher","unstructured":"Coles H, Laurent T, Henard C, Papadakis M, Ventresque A (2016) PIT: a practical mutation testing tool for java (demo). In: Proceedings of the 25th International Symposium on Software Testing and Analysis, ISSTA 2016, Saarbr\u00fccken, pp 449\u2013452. https:\/\/doi.org\/10.1145\/2931037.2948707","DOI":"10.1145\/2931037.2948707"},{"key":"9778_CR11","doi-asserted-by":"publisher","unstructured":"Delamaro ME, Maldonado JC, Vincenzi AMR (2001) Proteum\/IM 2.0: An Integrated Mutation Testing Environment, Springer US, Boston, chap Mutation Testing for the New Century, pp 91\u2013101. https:\/\/doi.org\/10.1007\/978-1-4757-5939-6_17","DOI":"10.1007\/978-1-4757-5939-6_17"},{"key":"9778_CR12","doi-asserted-by":"publisher","unstructured":"Delamaro ME, Offutt J, Ammann P (2014) Designing deletion mutation operators. In: Seventh IEEE international conference on software testing, verification and validation, ICST 2014, Cleveland, pp 11\u201320. https:\/\/doi.org\/10.1109\/ICST.2014.12","DOI":"10.1109\/ICST.2014.12"},{"key":"9778_CR13","doi-asserted-by":"publisher","first-page":"130","DOI":"10.1016\/j.infsof.2018.07.011","volume":"104","author":"P Delgado-P\u00e9rez","year":"2018","unstructured":"Delgado-P\u00e9rez P, Medina-Bulo I (2018) Search-based mutant selection for efficient test suite improvement: Evaluation and results. Inf Softw Technolo 104:130 \u2013 143. https:\/\/doi.org\/10.1016\/j.infsof.2018.07.011, http:\/\/www.sciencedirect.com\/science\/article\/pii\/S0950584918301551","journal-title":"Inf Softw Technolo"},{"issue":"4","key":"9778_CR14","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1109\/C-M.1978.218136","volume":"11","author":"RA DeMillo","year":"1978","unstructured":"DeMillo RA, Lipton RJ, Sayward FG (1978) Hints on test data selection: Help for the practicing programmer. IEEE Comput 11(4):34\u201341. https:\/\/doi.org\/10.1109\/C-M.1978.218136","journal-title":"IEEE Comput"},{"key":"9778_CR15","doi-asserted-by":"crossref","unstructured":"DeMillo RA, Guindi DS, King KN, McCracken WM, Offutt AJ (1988) An extended overview of the mothra software testing environment. In: Proceedings of the 2nd Workshop on Software Testing, Verification, and Analysis (TVA\u201988), Banff Alberta, pp 142\u2013151","DOI":"10.1109\/WST.1988.5369"},{"issue":"9","key":"9778_CR16","doi-asserted-by":"publisher","first-page":"900","DOI":"10.1109\/32.92910","volume":"17","author":"RA DeMillo","year":"1991","unstructured":"DeMillo RA, Offutt AJ (1991) Constraint-based automatic test data generation. IEEE Trans Softw Eng 17(9):900\u2013910. https:\/\/doi.org\/10.1109\/32.92910","journal-title":"IEEE Trans Softw Eng"},{"key":"9778_CR17","doi-asserted-by":"publisher","unstructured":"Deng L, Offutt J, Li N (2013) Empirical evaluation of the statement deletion mutation operator. In: Sixth IEEE international conference on software testing, verification and validation, ICST 2013, Luxembourg, pp 84\u201393. https:\/\/doi.org\/10.1109\/ICST.2013.20","DOI":"10.1109\/ICST.2013.20"},{"key":"9778_CR18","doi-asserted-by":"publisher","unstructured":"Ferrari FC, Pizzoleto AV, Offutt J (2018) A systematic review of cost reduction techniques for mutation testing: Preliminary results. In: 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp 1\u201310. https:\/\/doi.org\/10.1109\/ICSTW.2018.00021","DOI":"10.1109\/ICSTW.2018.00021"},{"issue":"3","key":"9778_CR19","doi-asserted-by":"publisher","first-page":"235","DOI":"10.1016\/S0164-1212(96)00154-9","volume":"38","author":"PG Frankl","year":"1997","unstructured":"Frankl PG, Weiss SN, Hu C (1997) All-uses vs mutation testing: an experimental comparison of effectiveness. J Syst Softw 38(3):235\u2013253. https:\/\/doi.org\/10.1016\/S0164-1212(96)00154-9","journal-title":"J Syst Softw"},{"issue":"6","key":"9778_CR20","doi-asserted-by":"publisher","first-page":"153","DOI":"10.1145\/291252.288298","volume":"23","author":"PG Frankl","year":"1998","unstructured":"Frankl PG, Iakounenko O (1998) Further empirical studies of test effectiveness. SIGSOFT Softw Eng Notes 23(6):153\u2013162. https:\/\/doi.org\/10.1145\/291252.288298","journal-title":"SIGSOFT Softw Eng Notes"},{"issue":"2","key":"9778_CR21","doi-asserted-by":"publisher","first-page":"278","DOI":"10.1109\/TSE.2011.93","volume":"38","author":"G Fraser","year":"2012","unstructured":"Fraser G, Zeller A (2012) Mutation-driven generation of unit tests and oracles. IEEE Trans Softw Eng 38(2):278\u2013292. https:\/\/doi.org\/10.1109\/TSE.2011.93","journal-title":"IEEE Trans Softw Eng"},{"issue":"4","key":"9778_CR22","doi-asserted-by":"publisher","first-page":"367","DOI":"10.1016\/S0167-9473(01)00065-2","volume":"38","author":"JH Friedman","year":"2002","unstructured":"Friedman JH (2002) Stochastic gradient boosting. Comput Stat Data Anal 38(4):367\u2013378","journal-title":"Comput Stat Data Anal"},{"key":"9778_CR23","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1016\/j.infsof.2016.05.001","volume":"81","author":"D Gong","year":"2017","unstructured":"Gong D, Zhang G, Yao X, Meng F (2017) Mutant reduction based on dominance relation for weak mutation testing. Inf Softw Technol 81:82\u201396. https:\/\/doi.org\/10.1016\/j.infsof.2016.05.001","journal-title":"Inf Softw Technol"},{"key":"9778_CR24","doi-asserted-by":"publisher","unstructured":"Hariri F, Shi A, Converse H, Khurshid S, Marinov D (2016) Evaluating the effects of compiler optimizations on mutation testing at the compiler IR level. In: 27th IEEE international symposium on software reliability engineering, ISSRE 2016, Ottawa, pp 105\u2013115. https:\/\/doi.org\/10.1109\/ISSRE.2016.51","DOI":"10.1109\/ISSRE.2016.51"},{"key":"9778_CR25","doi-asserted-by":"publisher","unstructured":"Henard C, Papadakis M, Harman M, Jia Y, Traon YL (2016) Comparing white-box and black-box test prioritization. In: Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, Austin, pp 523\u2013534. https:\/\/doi.org\/10.1145\/2884781.2884791","DOI":"10.1145\/2884781.2884791"},{"issue":"10","key":"9778_CR26","doi-asserted-by":"publisher","first-page":"1379","DOI":"10.1016\/j.infsof.2009.04.016","volume":"51","author":"Y Jia","year":"2009","unstructured":"Jia Y, Harman M (2009) Higher order mutation testing. Inf Softw Technol 51(10):1379\u20131393. https:\/\/doi.org\/10.1016\/j.infsof.2009.04.016","journal-title":"Inf Softw Technol"},{"key":"9778_CR27","doi-asserted-by":"publisher","unstructured":"Jiang L, Misherghi G, Su Z, Glondu S (2007) Deckard: Scalable and accurate tree-based detection of code clones. In: 29th International Conference on Software Engineering (ICSE\u201907), pp 96\u2013105. https:\/\/doi.org\/10.1109\/ICSE.2007.30","DOI":"10.1109\/ICSE.2007.30"},{"key":"9778_CR28","doi-asserted-by":"publisher","unstructured":"Just R, Jalali D, Ernst MD (2014a) Defects4j: a database of existing faults to enable controlled testing studies for java programs. In: International symposium on software testing and analysis, ISSTA \u201914, san Jose, pp 437\u2013440. https:\/\/doi.org\/10.1145\/2610384.2628055","DOI":"10.1145\/2610384.2628055"},{"key":"9778_CR29","doi-asserted-by":"publisher","unstructured":"Just R, Jalali D, Inozemtseva L, Ernst MD, Holmes R, Fraser G (2014b) Are mutants a valid substitute for real faults in software testing?. In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, (FSE-22), Hong Kong, 654\u2013665. https:\/\/doi.org\/10.1145\/2635868.2635929","DOI":"10.1145\/2635868.2635929"},{"key":"9778_CR30","doi-asserted-by":"publisher","unstructured":"Just R, Kurtz B, Ammann P (2017) Inferring mutant utility from program context. In: Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis, Santa Barbara, pp 284\u2013294. https:\/\/doi.org\/10.1145\/3092703.3092732","DOI":"10.1145\/3092703.3092732"},{"key":"9778_CR31","doi-asserted-by":"publisher","unstructured":"Kamei Y, Shihab E (2016) Defect prediction: Accomplishments and future challenges. In: Leaders of tomorrow symposium: Future of software engineering, FOSE@SANER 2016, Osaka, pp 33\u201345. https:\/\/doi.org\/10.1109\/SANER.2016.56","DOI":"10.1109\/SANER.2016.56"},{"key":"9778_CR32","unstructured":"Keck T (2016) Fastbdt: A speed-optimized and cache-friendly implementation of stochastic gradient-boosted decision trees for multivariate classification. arXiv:http:\/\/arXiv.org\/abs\/160906119"},{"key":"9778_CR33","doi-asserted-by":"publisher","unstructured":"Kintis M, Papadakis M, Malevris N (2010) Evaluating mutation testing alternatives: A collateral experiment. In: 17th asia pacific software engineering conference, APSEC 2010, Sydney, 300\u2013309. https:\/\/doi.org\/10.1109\/APSEC.2010.42","DOI":"10.1109\/APSEC.2010.42"},{"issue":"4","key":"9778_CR34","doi-asserted-by":"publisher","first-page":"308","DOI":"10.1109\/TSE.2017.2684805","volume":"44","author":"M Kintis","year":"2018","unstructured":"Kintis M, Papadakis M, Jia Y, Malevris N, Traon YL, Harman M (2018) Detecting trivial mutant equivalences via compiler optimisations. IEEE Trans Softw Eng 44(4):308\u2013333. https:\/\/doi.org\/10.1109\/TSE.2017.2684805","journal-title":"IEEE Trans Softw Eng"},{"issue":"1","key":"9778_CR35","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1109\/TSE.1986.6312924","volume":"12","author":"JC Knight","year":"1986","unstructured":"Knight JC, Leveson NG (1986) An experimental evaluation of the assumption of independence in multiversion programming. IEEE Trans Softw Eng 12(1):96\u2013109. https:\/\/doi.org\/10.1109\/TSE.1986.6312924","journal-title":"IEEE Trans Softw Eng"},{"key":"9778_CR36","doi-asserted-by":"publisher","unstructured":"Kurtz B, Ammann P, Offutt J, Delamaro ME, Kurtz M, G\u00f6k\u00e7e N (2016) Analyzing the validity of selective mutation with dominator mutants. In: Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2016, Seattle, pp 571\u2013582. https:\/\/doi.org\/10.1145\/2950290.2950322","DOI":"10.1145\/2950290.2950322"},{"key":"9778_CR37","unstructured":"Leveson NG (1995) Safeware - system safety and computers: a guide to preventing accidents and losses caused by technology. Addison-Wesley"},{"key":"9778_CR38","doi-asserted-by":"publisher","unstructured":"Li N, Praphamontripong U, Offutt J (2009) An experimental comparison of four unit test criteria: mutation, edge-pair, all-uses and prime path coverage. In: Mutation 2009, Denver, pp 220\u2013229. https:\/\/doi.org\/10.1109\/ICSTW.2009.30","DOI":"10.1109\/ICSTW.2009.30"},{"key":"9778_CR39","doi-asserted-by":"publisher","unstructured":"Ma Y, Offutt J, Kwon YR (2006) Mujava: a mutation system for java. In: 28th international conference on software engineering (ICSE 2006), Shanghai, 827\u2013830. https:\/\/doi.org\/10.1145\/1134425","DOI":"10.1145\/1134425"},{"key":"9778_CR40","doi-asserted-by":"crossref","unstructured":"Mao D, Chen L, Zhang L (2019) An extensive study on cross-project predictive mutation testing. In: 12th IEEE international conference on software testing, verification and validation, ICST 2019, Xian, pp 160\u2013171","DOI":"10.1109\/ICST.2019.00025"},{"key":"9778_CR41","doi-asserted-by":"crossref","unstructured":"Mathur AP (1991) Performance, effectiveness, and reliability issues in software testing. In: Proceedings of the 5th International Computer Software and Applications Conference (COMPSAC\u201979), Tokyo, pp 604\u2013605","DOI":"10.1109\/CMPSAC.1991.170248"},{"issue":"1","key":"9778_CR42","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1109\/TSE.2007.256941","volume":"33","author":"T Menzies","year":"2007","unstructured":"Menzies T, Greenwald J, Frank A (2007) Data mining static code attributes to learn defect predictors. IEEE Trans Softw Eng 33(1):2\u201313. https:\/\/doi.org\/10.1109\/TSE.2007.256941","journal-title":"IEEE Trans Softw Eng"},{"issue":"5","key":"9778_CR43","doi-asserted-by":"publisher","first-page":"429","DOI":"10.1109\/TSE.2014.2371458","volume":"41","author":"S Mirshokraie","year":"2015","unstructured":"Mirshokraie S, Mesbah A, Pattabiraman K (2015) Guided mutation testing for javascript web applications. IEEE Trans Softw Eng 41(5):429\u2013444. https:\/\/doi.org\/10.1109\/TSE.2014.2371458","journal-title":"IEEE Trans Softw Eng"},{"issue":"4","key":"9778_CR44","doi-asserted-by":"publisher","first-page":"205","DOI":"10.1002\/(SICI)1099-1689(199912)9:4<205::AID-STVR186>3.0.CO;2-X","volume":"9","author":"ES Mresa","year":"1999","unstructured":"Mresa ES, Bottaci L (1999) Efficiency of mutation operators and selective mutation strategies: An empirical study. Software Testing. Verif Reliab 9(4):205\u2013232","journal-title":"Verif Reliab"},{"key":"9778_CR45","unstructured":"Namin AS, Andrews JH, Murdoch DJ (2008) Sufficient mutation operators for measuring test effectiveness. In: Proceedings of the 30th International Conference on Software Engineering (ICSE\u201908), Leipzig, pp 351\u2013360"},{"issue":"5-7","key":"9778_CR46","doi-asserted-by":"publisher","first-page":"572","DOI":"10.1002\/stvr.1542","volume":"25","author":"AS Namin","year":"2015","unstructured":"Namin AS, Xue X, Rosas O, Sharma P (2015) Muranker: a mutant ranking tool. Softw Test Verif Reliab 25(5-7):572\u2013604. https:\/\/doi.org\/10.1002\/stvr.1542","journal-title":"Softw Test Verif Reliab"},{"key":"9778_CR47","doi-asserted-by":"crossref","unstructured":"Natekin A, Knoll A (2013) Gradient boosting machines, a tutorial. Frontiers in neurorobotics 7","DOI":"10.3389\/fnbot.2013.00021"},{"issue":"1","key":"9778_CR48","doi-asserted-by":"publisher","first-page":"80","DOI":"10.1109\/TSE.2011.124","volume":"39","author":"R Natella","year":"2013","unstructured":"Natella R, Cotroneo D, Dur\u00e3es J, Madeira H (2013) On fault representativeness of software fault injection. IEEE Trans Softw Eng 39(1):80\u201396. https:\/\/doi.org\/10.1109\/TSE.2011.124","journal-title":"IEEE Trans Softw Eng"},{"key":"9778_CR49","doi-asserted-by":"crossref","unstructured":"Offutt AJ, Rothermel G, Zapf C (1993) An experimental evaluation of selective mutation. In: Proceedings of the 15th International Conference on Software Engineering, ICSE \u201993. IEEE Computer Society Press, Los Alamitos, pp 100\u2013107. http:\/\/dl.acm.org\/citation.cfm?id=257572.257597","DOI":"10.1109\/ICSE.1993.346062"},{"issue":"3","key":"9778_CR50","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1002\/stvr.4370040303","volume":"4","author":"AJ Offutt","year":"1994","unstructured":"Offutt AJ, Craft WM (1994) Using compiler optimization techniques to detect equivalent mutants. Softw Test Verif Reliab 4(3):131\u2013154. https:\/\/doi.org\/10.1002\/stvr.4370040303","journal-title":"Softw Test Verif Reliab"},{"issue":"2","key":"9778_CR51","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1145\/227607.227610","volume":"5","author":"AJ Offutt","year":"1996","unstructured":"Offutt AJ, Lee A, Rothermel G, Untch RH, Zapf C (1996a) An experimental determination of sufficient mutant operators. ACM Trans Softw Eng Methodol 5(2):99\u2013118","journal-title":"ACM Trans Softw Eng Methodol"},{"issue":"2","key":"9778_CR52","doi-asserted-by":"publisher","first-page":"165","DOI":"10.1002\/(SICI)1097-024X(199602)26:2<165::AID-SPE5>3.0.CO;2-K","volume":"26","author":"AJ Offutt","year":"1996","unstructured":"Offutt AJ, Pan J, Tewary K, Zhang T (1996b) An experimental evaluation of data flow and mutation testing. Softw Pract Exper 26(2):165\u2013176. https:\/\/doi.org\/10.1002\/(SICI)1097-024X(199602)26:2<165::AID-SPE5>3.0.CO;2-K","journal-title":"Softw Pract Exper"},{"key":"9778_CR53","doi-asserted-by":"publisher","unstructured":"Palix N, Thomas G, Saha S, Calv\u00e8s C, Lawall J L, Muller G (2011) Faults in linux: ten years later. In: Proceedings of the 16th International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS 2011, Newport Beach, pp 305\u2013318. https:\/\/doi.org\/10.1145\/1950365.1950401","DOI":"10.1145\/1950365.1950401"},{"key":"9778_CR54","doi-asserted-by":"publisher","unstructured":"Papadakis M, Malevris N (2010a) Automatic mutation test case generation via dynamic symbolic execution. In: IEEE 21St international symposium on software reliability engineering, ISSRE 2010, San Jose, 121\u2013130. https:\/\/doi.org\/10.1109\/ISSRE.2010.38","DOI":"10.1109\/ISSRE.2010.38"},{"key":"9778_CR55","doi-asserted-by":"publisher","unstructured":"Papadakis M, Malevris N (2010b) An empirical evaluation of the first and second order mutation testing strategies. In: Third International Conference on Software Testing, Verification and Validation, ICST 2010. Workshops Proceedings, Paris, pp 90\u201399. https:\/\/doi.org\/10.1109\/ICSTW.2010.50","DOI":"10.1109\/ICSTW.2010.50"},{"key":"9778_CR56","doi-asserted-by":"publisher","unstructured":"Papadakis M, Jia Y, Harman M, Traon YL (2015) Trivial compiler equivalence: A large scale empirical study of a simple, fast and effective equivalent mutant detection technique. In: 37th IEEE\/ACM international conference on software engineering, ICSE 2015, Florence, pp 936\u2013946, https:\/\/doi.org\/10.1109\/ICSE.2015.103","DOI":"10.1109\/ICSE.2015.103"},{"key":"9778_CR57","doi-asserted-by":"publisher","unstructured":"Papadakis M, Henard C, Harman M, Jia Y, Traon YL (2016) Threats to the validity of mutation-based test assessment. In: Proceedings of the 25th International Symposium on Software Testing and Analysis, ISSTA 2016, Saarbru\u0307cken, pp 354\u2013365. https:\/\/doi.org\/10.1145\/2931037.2931040","DOI":"10.1145\/2931037.2931040"},{"key":"9778_CR58","doi-asserted-by":"crossref","unstructured":"Papadakis M, Kintis M, Zhang J, Jia Y, Traon YL, Harman M (2018a) Mutation testing advances: An analysis and survey. Advances. in Computers","DOI":"10.1016\/bs.adcom.2018.03.015"},{"key":"9778_CR59","doi-asserted-by":"crossref","unstructured":"Papadakis M, Shin D, Yoo S, Bae DH (2018b) are mutation scores correlated with real fault detection? a large scale empirical study on the relationship between mutants and real faults. In: Proceedings of the 40th International Conference on Software Engineering, ICSE","DOI":"10.1145\/3180155.3180183"},{"key":"9778_CR60","doi-asserted-by":"crossref","unstructured":"Papadakis M, Titcheu Chekam T, Le Traon Y (2018c) Mutant quality indicators. In: The 13th international workshop on mutation analysis, mutation","DOI":"10.1109\/ICSTW.2018.00025"},{"key":"9778_CR61","doi-asserted-by":"crossref","unstructured":"Petrovic G, Ivankovic M (2018) State of mutation testing at google. In: 40th IEEE\/ACM international conference on software engineering: Software engineering in practice track, ICSE-SEIP 2018, Gothenburg, Sweden","DOI":"10.1145\/3183519.3183521"},{"key":"9778_CR62","doi-asserted-by":"publisher","unstructured":"Ramler R, Wetzlmaier T, Klammer C (2017) An empirical study on the application of mutation testing for a safety-critical industrial software system. In: Proceedings of the Symposium on Applied Computing, SAC 2017, Marrakech, pp 1401\u20131408. https:\/\/doi.org\/10.1145\/3019612.3019830","DOI":"10.1145\/3019612.3019830"},{"issue":"10","key":"9778_CR63","doi-asserted-by":"publisher","first-page":"929","DOI":"10.1109\/32.962562","volume":"27","author":"G Rothermel","year":"2001","unstructured":"Rothermel G, Untch RH, Chu C, Harrold MJ (2001) Prioritizing test cases for regression testing. IEEE Trans Softw Eng 27(10):929\u2013948. https:\/\/doi.org\/10.1109\/32.962562","journal-title":"IEEE Trans Softw Eng"},{"issue":"5","key":"9778_CR64","doi-asserted-by":"publisher","first-page":"353","DOI":"10.1002\/stvr.1473","volume":"23","author":"D Schuler","year":"2013","unstructured":"Schuler D, Zeller A (2013) Covering and uncovering equivalent mutants. Softw Test. Verif Reliab 23(5):353\u2013374. https:\/\/doi.org\/10.1002\/stvr.1473","journal-title":"Verif Reliab"},{"key":"9778_CR65","unstructured":"SiR (2018) Software-artifact infrastructure repository. https:\/\/sir.unl.edu\/portal\/bios\/tcas.php, accessed: 2018-10-20"},{"key":"9778_CR66","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1016\/j.infsof.2016.02.006","volume":"81","author":"C Sun","year":"2017","unstructured":"Sun C, Xue F, Liu H, Zhang X (2017) A path-aware approach to mutant reduction in mutation testing. Inf Softw Technol 81:65\u201381. https:\/\/doi.org\/10.1016\/j.infsof.2016.02.006","journal-title":"Inf Softw Technol"},{"key":"9778_CR67","doi-asserted-by":"crossref","unstructured":"T Acree A, A Budd T, Demillo R, J Lipton R, G Sayward F (1979) Mutation analysis. Technical Report GIT-ICS-79\/08, pp 92","DOI":"10.21236\/ADA076575"},{"key":"9778_CR68","doi-asserted-by":"publisher","unstructured":"Tan SH, Yi J, Yulis Mechtaev S, Roychoudhury A (2017) Codeflaws: a programming competition benchmark for evaluating automated program repair tools. In: Proceedings of the 39th International Conference on Software Engineering, ICSE 2017, Buenos Aires, 2017 - Companion Volume, pp 180\u2013182. https:\/\/doi.org\/10.1109\/ICSE-C.2017.76","DOI":"10.1109\/ICSE-C.2017.76"},{"key":"9778_CR69","doi-asserted-by":"publisher","unstructured":"Titcheu Chekam T, Papadakis M, Le Traon Y, Harman M (2017) An empirical study on mutation, statement and branch coverage fault revelation that avoids the unreliable clean program assumption. In: Proceedings of the 39th International Conference on Software Engineering, ICSE 2017, Buenos Aires, pp 597\u2013608. https:\/\/doi.org\/10.1109\/ICSE.2017.61","DOI":"10.1109\/ICSE.2017.61"},{"key":"9778_CR70","doi-asserted-by":"publisher","unstructured":"Untch RH (2009) On reduced neighborhood mutation analysis using a single mutagenic operator. In: Proceedings of the 47th Annual Southeast Regional Conference, 2009, Clemson. https:\/\/doi.org\/10.1145\/1566445.1566540","DOI":"10.1145\/1566445.1566540"},{"issue":"2","key":"9778_CR71","first-page":"101","volume":"25","author":"A Vargha","year":"2000","unstructured":"Vargha A, Delaney HD (2000) A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong. Jrnl Educ Behav Stat 25 (2):101\u2013132","journal-title":"Jrnl Educ Behav Stat"},{"key":"9778_CR72","doi-asserted-by":"publisher","unstructured":"Wen M, Chen J, Wu R, Hao D, Cheung S (2018) Context-aware patch generation for better automated program repair. In: Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, pp 1\u201311. https:\/\/doi.org\/10.1145\/3180155.3180233","DOI":"10.1145\/3180155.3180233"},{"key":"9778_CR73","unstructured":"Wong WE (1993) On mutation and data flow. phdthesis Purdue University, West Lafayette"},{"key":"9778_CR74","doi-asserted-by":"publisher","unstructured":"Zhang L, Hou S, Hu J, Xie T, Mei H (2010) Is operator-based mutant selection superior to random mutant selection?. In: Proceedings of the 32nd ACM\/IEEE International Conference on Software Engineering - Volume 1, ICSE 2010, Cape Town, pp 435\u2013444. https:\/\/doi.org\/10.1145\/1806799.1806863","DOI":"10.1145\/1806799.1806863"},{"key":"9778_CR75","doi-asserted-by":"publisher","unstructured":"Zhang L, Marinov D, Khurshid S (2013) Faster mutation testing inspired by test prioritization and reduction. In: International symposium on software testing and analysis, ISSTA \u201913, Lugano, pp 235\u2013245. https:\/\/doi.org\/10.1145\/2483760.2483782","DOI":"10.1145\/2483760.2483782"},{"key":"9778_CR76","unstructured":"Zheng A (2015) Evaluating Machine Learning Models A Beginner\u2019s Guide to Key Concepts and Pitfalls. O\u2019Reilly Media, Inc. https:\/\/www.oreilly.com\/data\/free\/evaluating-machine-learning-models.csp"},{"issue":"3","key":"9778_CR77","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1016\/0164-1212(94)00098-0","volume":"31","author":"WE Wong","year":"1995","unstructured":"Wong WE, Mathur AP (1995a) Reducing the cost of mutation testing: an empirical study. J Syst Softw 31(3):185\u2013196. https:\/\/doi.org\/10.1016\/0164-1212(94)00098-0","journal-title":"J Syst Softw"},{"issue":"3","key":"9778_CR78","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1016\/0164-1212(94)00098-0","volume":"31","author":"WE Wong","year":"1995","unstructured":"Wong WE, Mathur AP (1995b) Reducing the cost of mutation testing: an empirical study. J Syst Softw 31(3):185\u2013196. https:\/\/doi.org\/10.1016\/0164-1212(94)00098-0","journal-title":"J Syst Softw"},{"key":"9778_CR79","doi-asserted-by":"publisher","unstructured":"Zhang J, Wang Z, Zhang L, Hao D, Zang L, Cheng S, Zhang L (2016) Predictive mutation testing. In: Proceedings of the 25th International Symposium on Software Testing and Analysis, ISSTA 2016, Saarbru\u0307cken, pp 342\u2013353. https:\/\/doi.org\/10.1145\/2931037.2931038","DOI":"10.1145\/2931037.2931038"},{"key":"9778_CR80","doi-asserted-by":"publisher","unstructured":"Zhang J, Zhang L, Harman M, Hao D, Jia Y, Zhang L (2018) Predictive mutation testing. IEEE Trans Softw Eng:1\u20131. https:\/\/doi.org\/10.1109\/TSE.2018.2809496","DOI":"10.1109\/TSE.2018.2809496"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-019-09778-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s10664-019-09778-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-019-09778-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,12,17]],"date-time":"2020-12-17T01:35:27Z","timestamp":1608168927000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s10664-019-09778-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,12,18]]},"references-count":80,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,1]]}},"alternative-id":["9778"],"URL":"https:\/\/doi.org\/10.1007\/s10664-019-09778-7","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,12,18]]},"assertion":[{"value":"18 December 2019","order":1,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}