{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,19]],"date-time":"2026-01-19T04:23:26Z","timestamp":1768796606011,"version":"3.49.0"},"reference-count":97,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2022,7,18]],"date-time":"2022-07-18T00:00:00Z","timestamp":1658102400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,7,18]],"date-time":"2022-07-18T00:00:00Z","timestamp":1658102400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000780","name":"European Commission","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100000780","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100011011","name":"Junta de Andaluc\u00eda","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100011011","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100011011","name":"Junta de Andaluc\u00eda","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100011011","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Spanish Government"},{"name":"Spanish Government"},{"DOI":"10.13039\/100009042","name":"Universidad de Sevilla","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100009042","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2022,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Mutation testing exploits artificial faults to measure the adequacy of test suites and guide their improvement. It has become an extremely popular testing technique as evidenced by the vast literature, numerous tools, and research events on the topic. Previous survey papers have successfully compiled the state of research, its evolution, problems, and challenges. However, the use of mutation testing in practice is still largely unexplored. In this paper, we report the results of a thorough study on the use of mutation testing in GitHub projects. Specifically, we first performed a search for mutation testing tools, 127 in total, and we automatically searched the GitHub repositories including evidence of their use. Then, we focused on the top ten most widely used tools, based on the previous results, and manually revised and classified over 3.5K GitHub active repositories importing them. Among other findings, we observed a recent upturn in interest and activity, with Infection (PHP), PIT (Java) and Humbug (PHP) being the most widely used mutation tools in recent years. The predominant use of mutation testing is development, followed by teaching and learning, and research projects, although with significant differences among mutation tools found in the literature\u2014less adopted and largely used in teaching and research\u2014and those found in GitHub only\u2014more popular and more widely used in development. Our work provides a new and encouraging perspective on the state of practice of mutation testing.<\/jats:p>","DOI":"10.1007\/s10664-022-10177-8","type":"journal-article","created":{"date-parts":[[2022,7,18]],"date-time":"2022-07-18T10:16:46Z","timestamp":1658139406000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":20,"title":["Mutation testing in the wild: findings from GitHub"],"prefix":"10.1007","volume":"27","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1473-0955","authenticated-orcid":false,"given":"Ana B.","family":"S\u00e1nchez","sequence":"first","affiliation":[]},{"given":"Pedro","family":"Delgado-P\u00e9rez","sequence":"additional","affiliation":[]},{"given":"Inmaculada","family":"Medina-Bulo","sequence":"additional","affiliation":[]},{"given":"Sergio","family":"Segura","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,7,18]]},"reference":[{"key":"10177_CR1","doi-asserted-by":"crossref","unstructured":"Acree AT, Budd TA, DeMillo RA, Lipton RJ, Sayward FG (1979) Mutation analysis. techreport GIT-ICS-79\/08, Georgia Institute of Technology, Atlanta, Georgia","DOI":"10.21236\/ADA076575"},{"key":"10177_CR2","doi-asserted-by":"crossref","unstructured":"Aggarwal CC, Zhai C (2012) Mining text data. Springer Science & Business Media","DOI":"10.1007\/978-1-4614-3223-4"},{"key":"10177_CR3","doi-asserted-by":"publisher","unstructured":"Ahmed S, Taj-Eddin IATF, Ismail MA (2020) MuHyb: A proposed mutation testing tool for hybrid mobile applications. In: Proceedings of the 2020 9th International Conference on Software and Information Engineering (ICSIE). ICSIE 2020. https:\/\/doi.org\/10.1145\/3436829.3436848. Association for Computing Machinery, New York, NY, USA, pp 67\u201372","DOI":"10.1145\/3436829.3436848"},{"key":"10177_CR4","unstructured":"Apache Maven Project. https:\/\/maven.apache.org\/. accessed in October 2021"},{"issue":"4","key":"10177_CR5","doi-asserted-by":"publisher","first-page":"1959","DOI":"10.1007\/s10664-017-9570-9","volume":"23","author":"A Arcuri","year":"2018","unstructured":"Arcuri A (2018) An experience report on applying software testing academic results in industry: we need usable automated test generation. Empir Softw Eng 23(4):1959\u20131981. https:\/\/doi.org\/10.1007\/s10664-017-9570-9","journal-title":"Empir Softw Eng"},{"key":"10177_CR6","unstructured":"Bamsurgeon. https:\/\/github.com\/adamewing\/bamsurgeon"},{"key":"10177_CR7","doi-asserted-by":"publisher","unstructured":"Bashir MB, Nadeem A (2018) An experimental tool for search-based mutation testing. In: 2018 international conference on frontiers of information technology (FIT). https:\/\/doi.org\/10.1109\/FIT.2018.00013, pp 30\u201334","DOI":"10.1109\/FIT.2018.00013"},{"key":"10177_CR8","doi-asserted-by":"crossref","unstructured":"Bashir M B, Nadeem A (2019) An evolutionary mutation testing system for Java programs: eMuJava. In: Intelligent computing-proceedings of the computing conference. Springer, pp 847\u2013865","DOI":"10.1007\/978-3-030-22868-2_58"},{"key":"10177_CR9","doi-asserted-by":"publisher","unstructured":"Beller M, Wong C, Bader J, Scott A, Machalica M, Chandra S, Meijer E (2021) What it would take to use mutation testing in industry \u2013 a study at Facebook. In: 2021 IEEE\/ACM 43rd international conference on software engineering: software engineering in practice (ICSE-SEIP). https:\/\/doi.org\/10.1109\/ICSE-SEIP52600.2021.00036. IEEE Computer Society, Los Alamitos, CA, USA, pp 268\u2013277","DOI":"10.1109\/ICSE-SEIP52600.2021.00036"},{"key":"10177_CR10","unstructured":"Bitbucket. https:\/\/bitbucket.org\/. accessed in October 2021"},{"key":"10177_CR11","doi-asserted-by":"publisher","unstructured":"Borges H, Hora A, Valente MT (2016) Understanding the factors that impact the popularity of GitHub repositories. In: 2016 IEEE international conference on software maintenance and evolution (ICSME). https:\/\/doi.org\/10.1109\/ICSME.2016.31, pp 334\u2013344","DOI":"10.1109\/ICSME.2016.31"},{"key":"10177_CR12","doi-asserted-by":"publisher","unstructured":"Brito G, Valente MT (2020) REST vs GraphQL: A controlled experiment. In: 2020 IEEE international conference on software architecture, ICSA 2020, Salvador, Brazil, March 16-20, 2020. https:\/\/doi.org\/10.1109\/ICSA47634.2020.00016. IEEE, pp 81\u201391","DOI":"10.1109\/ICSA47634.2020.00016"},{"key":"10177_CR13","doi-asserted-by":"publisher","first-page":"187","DOI":"10.1016\/j.jss.2018.05.010","volume":"143","author":"PC Ca\u00f1izares","year":"2018","unstructured":"Ca\u00f1izares P C, N\u00fa\u00f1ez A, Merayo M G (2018) Mutomvo: Mutation testing framework for simulated cloud and HPC environments. J Syst Softw 143:187\u2013207. https:\/\/doi.org\/10.1016\/j.jss.2018.05.010","journal-title":"J Syst Softw"},{"key":"10177_CR14","doi-asserted-by":"publisher","unstructured":"Chapman P, Xu D, Deng L, Xiong Y (2019) Deviant: A mutation testing tool for Solidity Smart Contracts. In: 2019 IEEE International Conference on Blockchain (Blockchain). https:\/\/doi.org\/10.1109\/Blockchain.2019.00050, pp 319\u2013324","DOI":"10.1109\/Blockchain.2019.00050"},{"key":"10177_CR15","doi-asserted-by":"publisher","unstructured":"Chekam T T, Papadakis M, Le Traon Y (2019) Mart: A mutant generation tool for LLVM. In: Proceedings of the 2019 27th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering. ESEC\/FSE 2019. https:\/\/doi.org\/10.1145\/3338906.3341180. Association for Computing Machinery, New York, NY, USA, pp 1080\u20131084","DOI":"10.1145\/3338906.3341180"},{"key":"10177_CR16","doi-asserted-by":"publisher","unstructured":"Chekam TT, Papadakis M, Traon YL (2020) Muteria: An extensible and flexible multi-criteria software testing framework. In: Proceedings of the IEEE\/ACM 1st international conference on automation of software test. AST \u201920. https:\/\/doi.org\/10.1145\/3387903.3389316. Association for Computing Machinery, New York, NY, USA, pp 97\u2013100","DOI":"10.1145\/3387903.3389316"},{"key":"10177_CR17","doi-asserted-by":"publisher","unstructured":"Coles H, Laurent T, Henard C, Papadakis M, Ventresque A (2016) PIT: A practical mutation testing tool for Java (demo). In: Proceedings of the 25th international symposium on software testing and analysis. ISSTA 2016. https:\/\/doi.org\/10.1145\/2931037.2948707. Association for Computing Machinery, New York, NY, USA, pp 449\u2013452","DOI":"10.1145\/2931037.2948707"},{"key":"10177_CR18","doi-asserted-by":"crossref","unstructured":"Cosentino V, Izquierdo JLC, Cabot J (2016) Findings from GitHub: Methods, datasets and limitations. In: 2016 IEEE\/ACM 13th working conference on mining software repositories (MSR), pp 137\u2013141","DOI":"10.1145\/2901739.2901776"},{"key":"10177_CR19","unstructured":"Cosmic-Ray. https:\/\/cosmic-ray.readthedocs.io\/en\/latest\/index.html"},{"issue":"7","key":"10177_CR20","doi-asserted-by":"publisher","first-page":"875","DOI":"10.1002\/spe.2312","volume":"45","author":"M Delahaye","year":"2015","unstructured":"Delahaye M, du Bousquet L (2015) Selecting a software engineering tool: lessons learnt from mutation analysis. Software: Practice and Experience 45(7):875\u2013891. https:\/\/doi.org\/10.1002\/spe.2312","journal-title":"Software: Practice and Experience"},{"key":"10177_CR21","doi-asserted-by":"publisher","unstructured":"Delamaro ME, Maldonado JC, Vincenzi A MR (2001) Proteum\/im 2.0: An integrated mutation testing environment. In: Wong WE (ed) Mutation Testing for the New Century. https:\/\/doi.org\/10.1007\/978-1-4757-5939-6\u2216_17. Springer US, Boston, MA, pp 91\u2013101","DOI":"10.1007\/978-1-4757-5939-6\u2216_17"},{"issue":"4","key":"10177_CR22","doi-asserted-by":"publisher","first-page":"1406","DOI":"10.1109\/TR.2018.2864678","volume":"67","author":"P Delgado-P\u00e9rez","year":"2018","unstructured":"Delgado-P\u00e9rez P, Habli I, Gregory S, Alexander R, Clark J, Medina-Bulo I (2018) Evaluation of mutation testing in a nuclear industry case study. IEEE Trans Reliab 67(4):1406\u20131419. https:\/\/doi.org\/10.1109\/TR.2018.2864678","journal-title":"IEEE Trans Reliab"},{"issue":"4","key":"10177_CR23","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1109\/C-M.1978.218136","volume":"11","author":"RA DeMillo","year":"1978","unstructured":"DeMillo RA, Lipton RJ, Sayward FG (1978) Hints on test data selection: Help for the practicing programmer. Computer 11(4):34\u201341. https:\/\/doi.org\/10.1109\/C-M.1978.218136","journal-title":"Computer"},{"key":"10177_CR24","unstructured":"DeMillo R A, Lipton R J, Sayward F G (1979) Program mutation: A new approach to program testing. In: Infotech State of the Art Report, Software Testing, pp 107\u2013126"},{"key":"10177_CR25","doi-asserted-by":"publisher","unstructured":"Denisov A, Pankevich S (2018) Mull it over: Mutation testing based on LLVM. In: 2018 IEEE international conference on software testing, verification and validation workshops (ICSTW). https:\/\/doi.org\/10.1109\/ICSTW.2018.00024, pp 25\u201331","DOI":"10.1109\/ICSTW.2018.00024"},{"key":"10177_CR26","doi-asserted-by":"publisher","unstructured":"Derezi\u0144ska A, Ha\u0142as K (2014) Analysis of mutation operators for the Python language. In: Proceedings of the Ninth international conference on dependability and complex systems DepCoS-RELCOMEX. June 30\u2013July 4, 2014, Brun\u00f3w, Poland. https:\/\/doi.org\/10.1007\/978-3-319-07013-1\u2216_15. Springer, pp 155\u2013164","DOI":"10.1007\/978-3-319-07013-1\u2216_15"},{"key":"10177_CR27","doi-asserted-by":"crossref","unstructured":"Efremidis A, Schmidt J, Krings S, K\u00f6rner P (2018) Measuring coverage of Prolog programs using mutation testing. In: International workshop on functional and constraint logic programming. Springer, pp 39\u201355","DOI":"10.1007\/978-3-030-16202-3_3"},{"key":"10177_CR28","doi-asserted-by":"publisher","unstructured":"Escobar-Vel\u00e1squez C, Riveros D, Linares-V\u00e1squez M (2020) MutAPK 2.0: A tool for reducing mutation testing effort of Android apps. In: Proceedings of the 28th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering. ESEC\/FSE 2020. https:\/\/doi.org\/10.1145\/3368089.3417942. Association for Computing Machinery, New York, NY, USA, pp 1611\u20131615","DOI":"10.1145\/3368089.3417942"},{"key":"10177_CR29","doi-asserted-by":"publisher","unstructured":"Escobar-Vel\u00e1squez C, Osorio-Ria\u00f1o M, Linares-V\u00e1squez M (2019) MutAPK: Source-codeless mutant generation for Android apps. In: 2019 34th IEEE\/ACM international conference on automated software engineering (ASE). https:\/\/doi.org\/10.1109\/ASE.2019.00109, pp 1090\u20131093","DOI":"10.1109\/ASE.2019.00109"},{"key":"10177_CR30","unstructured":"GitHub GraphQL API. https:\/\/docs.github.com\/en\/graphql. accessed in October 2021"},{"key":"10177_CR31","doi-asserted-by":"publisher","first-page":"85","DOI":"10.1016\/j.scico.2018.01.008","volume":"163","author":"P G\u00f3mez-Abajo","year":"2018","unstructured":"G\u00f3mez-Abajo P, Guerra E, de Lara J, Merayo M G (2018) A tool for domain-independent model mutation. Sci Comput Program 163:85\u201392. https:\/\/doi.org\/10.1016\/j.scico.2018.01.008","journal-title":"Sci Comput Program"},{"key":"10177_CR32","unstructured":"Go-Mutesting. https:\/\/github.com\/zimmski\/go-mutesting"},{"key":"10177_CR33","unstructured":"G\u00f3mez-Abajo P, Guerra E, de Lara J, Merayo M G (2018) Towards a model-driven engineering solution for language independent mutation testing. In: JISBD2018. http:\/\/hdl.handle.net\/11705\/JISBD\/2018\/052. SISTEDES"},{"key":"10177_CR34","doi-asserted-by":"publisher","unstructured":"Gonzalez D, Zimmermann T, Nagappan N (2020) The state of the ML-Universe: 10 years of artificial intelligence & machine learning software development on GitHub. In: Proceedings of the 17th international conference on mining software repositories. MSR \u201920. https:\/\/doi.org\/10.1145\/3379597.3387473. Association for Computing Machinery, New York, NY, USA, pp 431\u2013442","DOI":"10.1145\/3379597.3387473"},{"key":"10177_CR35","doi-asserted-by":"publisher","unstructured":"Gousios G, Spinellis D (2017) Mining software engineering data from GitHub. In: 2017 IEEE\/ACM 39th international conference on software engineering companion (ICSE-C). https:\/\/doi.org\/10.1109\/ICSE-C.2017.164https:\/\/doi.org\/10.1109\/ICSE-C.2017.164, pp 501\u2013502","DOI":"10.1109\/ICSE-C.2017.164 10.1109\/ICSE-C.2017.164"},{"key":"10177_CR36","doi-asserted-by":"crossref","unstructured":"Gousios G (2013) The GHTorent dataset and tool suite. In: Proceedings of the 10th working conference on mining software repositories. MSR \u201913. IEEE Press, pp 233\u2013236","DOI":"10.1109\/MSR.2013.6624034"},{"key":"10177_CR37","unstructured":"Gradle Build Tool. https:\/\/gradle.org\/. accessed in October 2021"},{"key":"10177_CR38","doi-asserted-by":"publisher","unstructured":"Groce A, Holmes J, Marinov D, Shi A, Zhang L (2018) An extensible, regular-expression-based tool for multi-language mutant generation. In: Proceedings of the 40th international conference on software engineering: companion proceeedings. ICSE \u201918. Association for Computing Machinery, New York, NY, USA, pp 25\u201328, DOI https:\/\/doi.org\/10.1145\/3183440.3183485, (to appear in print)","DOI":"10.1145\/3183440.3183485"},{"key":"10177_CR39","doi-asserted-by":"publisher","first-page":"78","DOI":"10.1016\/j.eswa.2018.09.003","volume":"116","author":"L Guti\u00e1rrez-Madro\u00f1al","year":"2019","unstructured":"Guti\u00e1rrez-Madro\u00f1al L, Medina-Bulo I, Dom\u00ednguez-Jim\u00e9nez JJ (2019) Evaluation of EPL mutation operators with the MuEPL mutation system. Expert Syst Appl 116:78\u201395.  https:\/\/doi.org\/10.1016\/j.eswa.2018.09.003","journal-title":"Expert Syst Appl"},{"key":"10177_CR40","doi-asserted-by":"publisher","unstructured":"Hariri F, Shi A, Fernando V, Mahmood S, Marinov D (2019) Comparing mutation testing at the levels of source code and compiler intermediate representation. In: 2019 12th IEEE conference on software testing, validation and verification (ICST). https:\/\/doi.org\/10.1109\/ICST.2019.00021, pp 114\u2013124","DOI":"10.1109\/ICST.2019.00021"},{"key":"10177_CR41","doi-asserted-by":"publisher","unstructured":"Hu Q, Ma L, Xie X, Yu B, Liu Y, Zhao J (2019) Deepmutation++: A mutation testing framework for deep learning systems. In: Proceedings of the 34th IEEE\/ACM international conference on automated software engineering. ASE \u201919. https:\/\/doi.org\/10.1109\/ASE.2019.00126. IEEE Press, pp 1158\u20131161","DOI":"10.1109\/ASE.2019.00126"},{"key":"10177_CR42","unstructured":"Humbug. https:\/\/github.com\/humbug\/humbug"},{"key":"10177_CR43","unstructured":"Infection. https:\/\/infection.github.io\/"},{"key":"10177_CR44","doi-asserted-by":"publisher","unstructured":"Irvine S A, Pavlinic T, Trigg L, Cleary J G, Inglis S, Utting M (2007) Jumble Java byte code to measure the effectiveness of unit tests. In: Testing: Academic and industrial conference practice and research techniques - MUTATION (TAICPART-MUTATION 2007). https:\/\/doi.org\/10.1109\/TAIC.PART.2007.38https:\/\/doi.org\/10.1109\/TAIC.PART.2007.38, pp 169\u2013175","DOI":"10.1109\/TAIC.PART.2007.38 10.1109\/TAIC.PART.2007.38"},{"key":"10177_CR45","doi-asserted-by":"publisher","first-page":"75","DOI":"10.1016\/j.procs.2020.11.009","volume":"178","author":"Y Ivanova","year":"2020","unstructured":"Ivanova Y, Khritankov A (2020) RegularMutator: A mutation testing tool for Solidity Smart Contracts. Procedia Computer Science 178:75\u201383. https:\/\/doi.org\/10.1016\/j.procs.2020.11.009. 9th International Young Scientists Conference in Computational Science, YSC2020, 05-12 September 2020","journal-title":"Procedia Computer Science"},{"key":"10177_CR46","unstructured":"Jester. http:\/\/jester.sourceforge.net\/"},{"issue":"5","key":"10177_CR47","doi-asserted-by":"publisher","first-page":"649","DOI":"10.1109\/TSE.2010.62","volume":"37","author":"Y Jia","year":"2011","unstructured":"Jia Y, Harman M (2011) An analysis and survey of the development of mutation testing. IEEE Trans Softw Eng 37(5):649\u2013678. https:\/\/doi.org\/10.1109\/TSE.2010.62","journal-title":"IEEE Trans Softw Eng"},{"key":"10177_CR48","doi-asserted-by":"publisher","unstructured":"Jovanovikj I, Weidmann N, Yigitbas E, Anjorin A, Sauer S, Engels G (2020) A model-driven mutation framework for validation of test case migration. In: Babur O, Denil J, Vogel-Heuser B (eds) Proceedings of the First international conference on systems modelling and management, ICSMM 2020 . https:\/\/doi.org\/10.1007\/978-3-030-58167-1\u2216_2. Springer","DOI":"10.1007\/978-3-030-58167-1\u2216_2"},{"issue":"1-2","key":"10177_CR49","doi-asserted-by":"publisher","first-page":"e1697","DOI":"10.1002\/stvr.1697","volume":"29","author":"R Just","year":"2019","unstructured":"Just R, Krinke J, Li N, Rojas JM (2019) Special issue on mutation testing and analysis. Software Testing, Verification and Reliability 29(1-2):e1697. https:\/\/doi.org\/10.1002\/stvr.1697","journal-title":"Software Testing, Verification and Reliability"},{"key":"10177_CR50","doi-asserted-by":"publisher","unstructured":"Just R (2014) The Major mutation framework: Efficient and scalable mutation analysis for Java. In: Proceedings of the 2014 international symposium on software testing and analysis. ISSTA 2014. https:\/\/doi.org\/10.1145\/2610384.2628053. Association for Computing Machinery, New York, NY, USA, pp 433\u2013436","DOI":"10.1145\/2610384.2628053"},{"key":"10177_CR51","doi-asserted-by":"publisher","unstructured":"Just R, Jalali D, Inozemtseva L, Ernst M D, Holmes R, Fraser G (2014) Are mutants a valid substitute for real faults in software testing?. FSE 2014. https:\/\/doi.org\/10.1145\/2635868.2635929. Association for Computing Machinery, New York, NY, USA, pp 654\u2013665","DOI":"10.1145\/2635868.2635929"},{"key":"10177_CR52","doi-asserted-by":"publisher","unstructured":"Kalliamvakou E, Gousios G, Blincoe K, Singer L, German D M, Damian D (2014) The promises and perils of mining GitHub. In: Proceedings of the 11th working conference on mining software repositories. MSR 2014. https:\/\/doi.org\/10.1145\/2597073.2597074. Association for Computing Machinery, New York, NY, USA, pp 92\u2013101","DOI":"10.1145\/2597073.2597074"},{"issue":"4","key":"10177_CR53","doi-asserted-by":"publisher","first-page":"2426","DOI":"10.1007\/s10664-017-9582-5","volume":"23","author":"M Kintis","year":"2018","unstructured":"Kintis M, Papadakis M, Papadopoulos A, Valvis E, Malevris N, Le Traon Y (2018) How effective are mutation testing tools? An empirical analysis of java mutation testing tools with manual analysis and real faults. Empir Softw Eng 23 (4):2426\u20132463. https:\/\/doi.org\/10.1007\/s10664-017-9582-5","journal-title":"Empir Softw Eng"},{"key":"10177_CR54","doi-asserted-by":"publisher","unstructured":"Li Z, Wu H, Xu J, Wang X, Zhang L, Chen Z (2019) Musc: A tool for mutation testing of Ethereum smart contract. In: 2019 34th IEEE\/ACM international conference on automated software engineering (ASE). https:\/\/doi.org\/10.1109\/ASE.2019.00136, pp 1198\u20131201","DOI":"10.1109\/ASE.2019.00136"},{"key":"10177_CR55","doi-asserted-by":"publisher","unstructured":"Liu M, Hong W, Pan W, Feng C, Chen Z, Wang J (2020) Styx: A data-oriented mutation framework to improve the robustness of DNN. In: Proceedings of the 35th IEEE\/ACM international conference on automated software engineering. ASE \u201920. https:\/\/doi.org\/10.1145\/3324884.3418903. Association for Computing Machinery, New York, NY, USA, pp 1260\u20131261","DOI":"10.1145\/3324884.3418903"},{"key":"10177_CR56","doi-asserted-by":"publisher","unstructured":"Luna E, Ariss O E (2018) Edroid: A mutation tool for Android apps. In: 2018 6th international conference in software engineering research and innovation (CONISOFT). https:\/\/doi.org\/10.1109\/CONISOFT.2018.8645883https:\/\/doi.org\/10.1109\/CONISOFT.2018.8645883, pp 99\u2013108","DOI":"10.1109\/CONISOFT.2018.8645883 10.1109\/CONISOFT.2018.8645883"},{"key":"10177_CR57","doi-asserted-by":"publisher","unstructured":"Ma L, Zhang F, Sun J, Xue M, Li B, Juefei-Xu F, Xie C, Li L, Liu Y, Zhao J, Wang Y (2018) Deepmutation: Mutation testing of deep learning systems. In: 2018 IEEE 29th international symposium on software reliability engineering (ISSRE). https:\/\/doi.org\/10.1109\/ISSRE.2018.00021, pp 100\u2013111","DOI":"10.1109\/ISSRE.2018.00021"},{"issue":"2","key":"10177_CR58","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1002\/stvr.308","volume":"15","author":"Y-S Ma","year":"2005","unstructured":"Ma Y-S, Offutt J, Kwon Y R (2005) MuJava: an automated class mutation system. Software Testing, Verification and Reliability 15 (2):97\u2013133. https:\/\/doi.org\/10.1002\/stvr.308","journal-title":"Software Testing, Verification and Reliability"},{"key":"10177_CR59","doi-asserted-by":"publisher","unstructured":"M\u00e1rki A, Lindstr\u00f6m B (2017) Mutation tools for Java. In: Proceedings of the symposium on applied computing. SAC \u201917. https:\/\/doi.org\/10.1145\/3019612.3019825. Association for Computing Machinery, New York, NY, USA, pp 1364\u20131415","DOI":"10.1145\/3019612.3019825"},{"key":"10177_CR60","unstructured":"Mombach1 T, Valente M T (2018) GitHub REST API vs GHTorrent vs GitHub Archive: A comparative study. https:\/\/homepages.dcc.ufmg.br\/mtov\/pub\/2018-vem-thais.pdf"},{"key":"10177_CR61","unstructured":"Momigliano A, Ornaghi M (2019) The blame game for property-based testing.. In: CILC, pp 4\u201313"},{"key":"10177_CR62","unstructured":"HtmlUnit. https:\/\/htmlunit.sourceforge.io. accessed in October 2021"},{"key":"10177_CR63","unstructured":"MuJava download. https:\/\/cs.gmu.edu\/~offutt\/mujava\/#Links. accessed in October 2021"},{"key":"10177_CR64","unstructured":"MuJava. https:\/\/github.com\/jeffoffutt\/muJava. accessed in October 2021"},{"key":"10177_CR65","unstructured":"Mull releases, https:\/\/github.com\/mull-project\/mull\/releases. accessed in October 2021"},{"key":"10177_CR66","unstructured":"Mutant. https:\/\/github.com\/mbj\/mutant. accessed in October 2021"},{"key":"10177_CR67","unstructured":"Muter. https:\/\/github.com\/muter-mutation-testing\/muter"},{"key":"10177_CR68","unstructured":"Mutmut. https:\/\/github.com\/boxed\/mutmut. accessed in October 2021"},{"key":"10177_CR69","unstructured":"MutPy. https:\/\/github.com\/mutpy\/mutpy. accessed in October 2021"},{"key":"10177_CR70","unstructured":"Ngambenchawong C, Suwannasart T (2019) A weak mutation testing framework for BPMN. In: Proceedings of the international multiconference of engineers and computer scientists, IMECS 2019, pp 13\u201315"},{"key":"10177_CR71","first-page":"34","volume-title":"Mutation 2000: Uniting the orthogonal","author":"AJ Offutt","year":"2001","unstructured":"Offutt A J, Untch R H (2001) Mutation 2000: Uniting the orthogonal. Kluwer Academic Publishers, USA, pp 34\u201344"},{"issue":"C","key":"10177_CR72","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.infsof.2016.08.003","volume":"81","author":"M Papadakis","year":"2017","unstructured":"Papadakis M, Just R (January 2017) Special issue on mutation testing. Inf Softw Technol 81(C):1\u20132. https:\/\/doi.org\/10.1016\/j.infsof.2016.08.003","journal-title":"Inf Softw Technol"},{"key":"10177_CR73","doi-asserted-by":"publisher","unstructured":"Papadakis M, Kintis M, Zhang J, Jia Y, Traon Y L, Harman M (2019) Chapter six - mutation testing advances: An analysis and survey. Elsevier, pp 275\u2013378. https:\/\/doi.org\/10.1016\/bs.adcom.2018.03.015","DOI":"10.1016\/bs.adcom.2018.03.015"},{"key":"10177_CR74","doi-asserted-by":"publisher","unstructured":"Petrovi\u0107 G, Ivankovi\u0107 M, Fraser G, Just R (2021) Does mutation testing improve testing practices?. In: 2021 IEEE\/ACM 43rd international conference on software engineering (ICSE). https:\/\/doi.org\/10.1109\/ICSE43902.2021.00087. IEEE Computer Society, Los Alamitos, CA, USA, pp 910\u2013921","DOI":"10.1109\/ICSE43902.2021.00087"},{"key":"10177_CR75","doi-asserted-by":"publisher","unstructured":"Petrovic G, Ivankovic M, Fraser G, Just R (2021) Practical mutation testing at scale: A view from Google. IEEE Trans Softw Eng, pp 1\u20131. https:\/\/doi.org\/10.1109\/TSE.2021.3107634","DOI":"10.1109\/TSE.2021.3107634"},{"key":"10177_CR76","doi-asserted-by":"publisher","unstructured":"Phan DL, Kim Y, Kim M (2018) Music: Mutation analysis tool with high configurability and extensibility. In: 2018 IEEE international conference on software testing, verification and validation workshops (ICSTW). https:\/\/doi.org\/10.1109\/ICSTW.2018.00026, pp 40\u201346","DOI":"10.1109\/ICSTW.2018.00026"},{"key":"10177_CR77","doi-asserted-by":"publisher","first-page":"102418","DOI":"10.1016\/j.scico.2020.102418","volume":"191","author":"P Pinheiro","year":"2020","unstructured":"Pinheiro P, Viana JC, Ribeiro M, Fernandes L, Ferrari F, Gheyi R, Fonseca B (2020) Mutating code annotations: An empirical evaluation on Java and C# programs. Sci Comput Program 191:102418. https:\/\/doi.org\/10.1016\/j.scico.2020.102418","journal-title":"Sci Comput Program"},{"key":"10177_CR78","unstructured":"PIT: installation with Maven and Gradle. https:\/\/pitest.org\/quickstart\/maven\/, https:\/\/gradle-pitest-plugin.solidsoft.info\/. accessed in October 2021"},{"key":"10177_CR79","unstructured":"PIT. https:\/\/github.com\/hcoles\/pitest. accessed in October 2021"},{"key":"10177_CR80","doi-asserted-by":"publisher","first-page":"110388","DOI":"10.1016\/j.jss.2019.07.100","volume":"157","author":"AV Pizzoleto","year":"2019","unstructured":"Pizzoleto AV, Ferrari FC, Offutt J, Fernandes L, Ribeiro M (2019) A systematic literature review of techniques and metrics to reduce the cost of mutation testing. J Syst Softw 157:110388. https:\/\/doi.org\/10.1016\/j.jss.2019.07.100","journal-title":"J Syst Softw"},{"key":"10177_CR81","unstructured":"Pujari A K (2001) Data mining techniques. Universities press"},{"key":"10177_CR82","doi-asserted-by":"publisher","unstructured":"Rodrigues E, Montecchi L, Ceccarelli A (2020) Model-driven fault injection in Java source code. In: 2020 IEEE 31st international symposium on software reliability engineering (ISSRE). https:\/\/doi.org\/10.1109\/ISSRE5003.2020.00046, pp 414\u2013425","DOI":"10.1109\/ISSRE5003.2020.00046"},{"key":"10177_CR83","doi-asserted-by":"publisher","unstructured":"Rodr\u00edguez-Baquero D, Linares-V\u00e1squez M (2018) Mutode: Generic JavaScript and Node.Js mutation testing tool. In: Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis. ISSTA 2018. https:\/\/doi.org\/10.1145\/3213846.3229504. Association for Computing Machinery, New York, NY, USA, pp 372\u2013375","DOI":"10.1145\/3213846.3229504"},{"key":"10177_CR84","doi-asserted-by":"publisher","unstructured":"Sadath L, Nair R (2019) Oncological inspired techniques for intelligent software testing. In: 2019 amity international conference on artificial intelligence (AICAI). https:\/\/doi.org\/10.1109\/AICAI.2019.8701251, pp 327\u2013333","DOI":"10.1109\/AICAI.2019.8701251"},{"key":"10177_CR85","doi-asserted-by":"publisher","unstructured":"Siavashi F, Truscan D, Vain J (2018) Vulnerability assessment of web services with model-based mutation testing. In: 2018 IEEE international conference on software quality, reliability and security (QRS). https:\/\/doi.org\/10.1109\/QRS.2018.00043, pp 301\u2013312","DOI":"10.1109\/QRS.2018.00043"},{"key":"10177_CR86","unstructured":"Stryker-NET. https:\/\/stryker-mutator.io\/docs\/stryker-net\/Introduction"},{"key":"10177_CR87","unstructured":"StrykerJS. https:\/\/stryker-mutator.io\/docs\/stryker-js\/introduction"},{"issue":"12","key":"10177_CR88","doi-asserted-by":"publisher","first-page":"5406","DOI":"10.35940\/ijitee.L3789.1081219","volume":"8","author":"S Suguna Mallika","year":"2019","unstructured":"Suguna Mallika S, Rajya Lakshmi D (2019) MUTWEB-A testing tool for performing mutation testing of Java and servlet based web applications. International Journal of Innovative Technology and Exploring Engineering 8(12):5406\u20135413","journal-title":"International Journal of Innovative Technology and Exploring Engineering"},{"key":"10177_CR89","doi-asserted-by":"publisher","unstructured":"\u201cSupplementary material of the paper\u201d? [Online]. Available at https:\/\/doi.org\/10.5281\/zenodo.5713585","DOI":"10.5281\/zenodo.5713585"},{"key":"10177_CR90","unstructured":"The 17th International Workshop on Mutation Analysis - MUTATION 2022. https:\/\/icst2022.vrain.upv.es\/home\/mutation-2022\/"},{"issue":"3","key":"10177_CR91","doi-asserted-by":"publisher","first-page":"80","DOI":"10.1109\/MS.2010.79","volume":"27","author":"MP Usaola","year":"2010","unstructured":"Usaola M P, Mateo P R (2010) Mutation testing cost reduction techniques: A survey. IEEE Softw 27(3):80\u201386. https:\/\/doi.org\/10.1109\/MS.2010.79","journal-title":"IEEE Softw"},{"key":"10177_CR92","doi-asserted-by":"publisher","unstructured":"Uzunbayir S, Kurtel K (2019) An analysis on mutation testing tools for C# programming language. In: 2019 4th international conference on computer science and engineering (UBMK). https:\/\/doi.org\/10.1109\/UBMK.2019.8907222, pp 439\u2013443","DOI":"10.1109\/UBMK.2019.8907222"},{"key":"10177_CR93","doi-asserted-by":"publisher","unstructured":"Van Phol L, Binh N T (2020) Optimizing mutant generation for Lustre programs with multi-threading. In: 2020 5th international conference on innovative technologies in intelligent systems and industrial applications (CITISIA). https:\/\/doi.org\/10.1109\/CITISIA50690.2020.9397490, pp 1\u20135","DOI":"10.1109\/CITISIA50690.2020.9397490"},{"key":"10177_CR94","doi-asserted-by":"publisher","unstructured":"Wang K, Sullivan A, Khurshid S (2018) MuAlloy: A mutation testing framework for Alloy. In: Proceedings of the 40th international conference on software engineering: companion proceeedings. ICSE \u201918. https:\/\/doi.org\/10.1145\/3183440.3183488https:\/\/doi.org\/10.1145\/3183440.3183488. Association for Computing Machinery, New York, NY, USA, pp 29\u201332","DOI":"10.1145\/3183440.3183488 10.1145\/3183440.3183488"},{"key":"10177_CR95","doi-asserted-by":"publisher","DOI":"10.1016\/B978-0-12-804291-5.00014-3","volume-title":"Data mining (fourth edition)","year":"2017","unstructured":"Witten IH, Frank E, Hall MA, Pal CJ (eds) (2017) Data mining (fourth edition), 4th Edn. Morgan Kaufmann, Burlington. https:\/\/doi.org\/10.1016\/B978-0-12-804291-5.00014-3","edition":"4th Edn"},{"key":"10177_CR96","unstructured":"WODEL: Get started. https:\/\/github.com\/gomezabajo\/Wodel\/wiki\/Get-Started. accessed in October 2021"},{"key":"10177_CR97","doi-asserted-by":"publisher","unstructured":"Xiong Y, Meng Z, Shen B, Yin W (2017) Mining developer behavior across GitHub and StackOverflow. In: The 29th international conference on software engineering and knowledge engineering. https:\/\/doi.org\/10.18293\/SEKE2017-062, pp 578\u2013583","DOI":"10.18293\/SEKE2017-062"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-022-10177-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-022-10177-8\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-022-10177-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,9,26]],"date-time":"2022-09-26T08:17:46Z","timestamp":1664180266000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-022-10177-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,18]]},"references-count":97,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2022,11]]}},"alternative-id":["10177"],"URL":"https:\/\/doi.org\/10.1007\/s10664-022-10177-8","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7,18]]},"assertion":[{"value":"16 May 2022","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 July 2022","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}],"article-number":"132"}}