{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,2]],"date-time":"2025-11-02T04:46:53Z","timestamp":1762058813822,"version":"build-2065373602"},"reference-count":53,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2022,7,7]],"date-time":"2022-07-07T00:00:00Z","timestamp":1657152000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Mutation testing is an effective, yet costly, testing approach, as it requires generating and running large numbers of faulty programs, called mutants. Mutation testing also suffers from a fundamental problem, which is having a large percentage of equivalent mutants. These are mutants that produce the same output as the original program, and therefore, cannot be detected. Higher-order mutation is a promising approach that can produce hard-to-detect faulty programs called subtle mutants, with a low percentage of equivalent mutants. Subtle higher-order mutants contribute a small set of the large space of mutants which grows even larger as the order of mutation becomes higher. In this paper, we developed a genetic algorithm for finding subtle higher-order mutants. The proposed approach uses a new mechanism in the crossover phase and uses five selection techniques to select mutants that go to the next generation in the genetic algorithm. We implemented a tool, called GaSubtle that automates the process of creating subtle mutants. We evaluated the proposed approach by using 10 subject programs. Our evaluation shows that the proposed crossover generates more subtle mutants than the technique used in a previous genetic algorithm with less execution time. Results vary on the selection strategies, suggesting a dependency relation with the tested code.<\/jats:p>","DOI":"10.3390\/info13070327","type":"journal-article","created":{"date-parts":[[2022,7,7]],"date-time":"2022-07-07T07:51:56Z","timestamp":1657180316000},"page":"327","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["GaSubtle: A New Genetic Algorithm for Generating Subtle Higher-Order Mutants"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4464-8688","authenticated-orcid":false,"given":"Fadi","family":"Wedyan","sequence":"first","affiliation":[{"name":"Department of Software Engineering, Faculty of Prince Al-Hussein bin Abdullah II of Information Technology, The Hashemite University, P.O. Box 330127, Zarqa 13133, Jordan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Abdullah","family":"Al-Shishani","sequence":"additional","affiliation":[{"name":"Department of Software Engineering, Faculty of Prince Al-Hussein bin Abdullah II of Information Technology, The Hashemite University, P.O. Box 330127, Zarqa 13133, Jordan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4403-3846","authenticated-orcid":false,"given":"Yaser","family":"Jararweh","sequence":"additional","affiliation":[{"name":"Department of Mathematics and Computer Science, Graduate School of Liberal Arts, McAnulty College, Duquesne University, Pittsburgh, PA 15282, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,7,7]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1109\/C-M.1978.218136","article-title":"Hints on test data selection: Help for the practicing programmer","volume":"11","author":"DeMillo","year":"1978","journal-title":"Computer"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1109\/TSE.1977.231145","article-title":"Testing programs with the aid of a compiler","volume":"SE-3","author":"Hamlet","year":"1977","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1016\/S0164-1212(96)00154-9","article-title":"All-uses vs mutation testing: An experimental comparison of effectiveness","volume":"38","author":"Frankl","year":"1997","journal-title":"J. Syst. Softw."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"608","DOI":"10.1109\/TSE.2006.83","article-title":"Using mutation analysis for assessing and comparing testing coverage criteria","volume":"32","author":"Andrews","year":"2006","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1379","DOI":"10.1016\/j.infsof.2009.04.016","article-title":"Higher Order Mutation Testing","volume":"51","author":"Jia","year":"2009","journal-title":"Inf. Softw. Technol."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1007\/BF00625279","article-title":"Two notions of correctness and their relation to testing","volume":"18","author":"Budd","year":"1982","journal-title":"Acta Inform."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"353","DOI":"10.1002\/stvr.1473","article-title":"Covering and uncovering equivalent mutants","volume":"23","author":"Schuler","year":"2013","journal-title":"Softw. Test. Verif. Reliab."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1002\/(SICI)1099-1689(199912)9:4<205::AID-STVR186>3.0.CO;2-X","article-title":"Efficiency of mutation operators and selective mutation strategies: An empirical study","volume":"9","author":"Mresa","year":"1999","journal-title":"Softw. Test. Verif. Reliab."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"337","DOI":"10.1109\/32.286422","article-title":"An empirical evaluation of weak mutation","volume":"20","author":"Offutt","year":"1994","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_10","unstructured":"Budd, T.A. (1980). Mutation Analysis of Program Test Data. [Ph.D. Thesis, Yale University]."},{"key":"ref_11","unstructured":"Hussain, S. (2008). Mutation Clustering. [Master\u2019s Thesis, King\u2019s College London]."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"371","DOI":"10.1109\/TSE.1982.235571","article-title":"Weak mutation testing and completeness of test sets","volume":"SE-8","author":"Howden","year":"1982","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Baldwin, D., and Sayward, F. (1979). Heuristics for Determining Equivalence of Program Mutations, School of Computer Science. Technical Report.","DOI":"10.21236\/ADA071795"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1002\/(SICI)1099-1689(199709)7:3<165::AID-STVR143>3.0.CO;2-U","article-title":"Automatically detecting equivalent mutants and infeasible paths","volume":"7","author":"Offutt","year":"1997","journal-title":"Softw. Test. Verif. Reliab."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"233","DOI":"10.1002\/(SICI)1099-1689(199912)9:4<233::AID-STVR191>3.0.CO;2-3","article-title":"Using program slicing to assist in the detection of equivalent mutants","volume":"9","author":"Hierons","year":"1999","journal-title":"Softw. Test. Verif. Reliab."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Grun, B.J., Schuler, D., and Andreas, Z. (2009, January 1\u20134). The impact of equivalent mutants. Proceedings of the Software Testing, Verification and Validation Workshops, Denver, CO, USA.","DOI":"10.1109\/ICSTW.2009.37"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Adamopoulos, K., Harman, M., and Hierons, R.M. (2004, January 26\u201330). How to overcome the equivalent mutant problem and achieve tailored selective mutation using co-evolution. Proceedings of the Genetic and Evolutionary Computation Conference, Seattle, WA, USA.","DOI":"10.1007\/978-3-540-24855-2_155"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"511","DOI":"10.1109\/TSE.2005.74","article-title":"Toward understanding the rhetoric of small source code changes","volume":"31","author":"Purushothaman","year":"2005","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Gopinath, R., Jensen, C., and Groce, A. (2014, January 3\u20136). Mutations: How close are they to real faults?. Proceedings of the Software Reliability Engineering (ISSRE), Naples, Italy.","DOI":"10.1109\/ISSRE.2014.40"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"2416","DOI":"10.1016\/j.jss.2010.07.027","article-title":"Efficient Multi-objective Higher Order Mutation Testing with Genetic Programming","volume":"83","author":"Langdon","year":"2010","journal-title":"J. Syst. Softw."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Omar, E., and Ghosh, S. (2012, January 27\u201330). An exploratory study of higher order mutation testing in aspect-oriented programming. Proceedings of the Software Reliability Engineering (ISSRE), Dallas, TX, USA.","DOI":"10.1109\/ISSRE.2012.6"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"570","DOI":"10.1109\/TSE.2012.39","article-title":"Validating second-order mutation at system level","volume":"39","author":"Mateo","year":"2013","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Omar, E., Ghosh, S., and Whitley, D. (2013, January 4\u20137). Constructing subtle higher order mutants for Java and AspectJ programs. Proceedings of the Software Reliability Engineering (ISSRE), Pasadena, CA, USA.","DOI":"10.1109\/ISSRE.2013.6698887"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"900","DOI":"10.1016\/j.infsof.2011.12.001","article-title":"On generating mutants for AspectJ programs","volume":"54","author":"Wedyan","year":"2012","journal-title":"Inf. Softw. Technol."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Jia, Y., and Harman, M. (2008, January 28\u201329). Constructing subtle faults using higher order mutation testing. Proceedings of the Source Code Analysis and Manipulation, Beijing, China.","DOI":"10.1109\/SCAM.2008.36"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Kintis, M., Papadakis, M., and Malevris, N. (2012, January 17\u201321). Isolating first order equivalent mutants via second order mutation. Proceedings of the Software Testing, Verification and Validation (ICST), Montreal, QC, Canada.","DOI":"10.1109\/ICST.2012.160"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1016\/j.jss.2019.04.031","article-title":"A systematic mapping study on higher order mutation testing","volume":"154","author":"Vergilio","year":"2019","journal-title":"J. Syst. Softw."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1016\/j.cosrev.2017.06.001","article-title":"Higher order mutation testing: A systematic literature review","volume":"25","author":"Ghiduk","year":"2017","journal-title":"Comput. Sci. Rev."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"110388","DOI":"10.1016\/j.jss.2019.07.100","article-title":"A systematic literature review of techniques and metrics to reduce the cost of mutation testing","volume":"157","author":"Pizzoleto","year":"2019","journal-title":"J. Syst. Softw."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Lima, J.A.P., and Vergilio, S.R. (2018, January 17\u201321). Search-based higher order mutation testing: A mapping study. Proceedings of the III Brazilian Symposium on Systematic and Automated Software Testing, Sao Carlos, Brazil.","DOI":"10.1145\/3266003.3266013"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1002\/stvr.392","article-title":"Decreasing the Cost of Mutation Testing with Second-order Mutants","volume":"19","author":"Polo","year":"2009","journal-title":"Softw. Test. Verif. Reliab."},{"key":"ref_32","first-page":"416","article-title":"Genetic Algorithms for Multiobjective Optimization: FormulationDiscussion and Generalization","volume":"93","author":"Fonseca","year":"1993","journal-title":"InIcga"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"182","DOI":"10.1109\/4235.996017","article-title":"A fast and elitist multiobjective genetic algorithm: NSGA-II","volume":"6","author":"Deb","year":"2002","journal-title":"IEEE Trans. Evol. Comput."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Ma, Y.S., Offutt, J., and Kwon, Y.R. (2006, January 20\u201328). MuJava: A Mutation System for Java. Proceedings of the 28th International Conference on Software Engineering, Shanghai, China.","DOI":"10.1145\/1134285.1134425"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Delamare, R., Baudry, B., and Le Traon, Y. (2009, January 1\u20134). AjMutator: A tool for the mutation analysis of AspectJ pointcut descriptors. Proceedings of the Software Testing, Verification and Validation Workshops, Denver, CO, USA.","DOI":"10.1109\/ICSTW.2009.41"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Omar, E., Ghosh, S., and Whitley, D. (2014, January 12\u201316). Comparing search techniques for finding subtle higher order mutants. Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation, Vancouver, BC, Canada.","DOI":"10.1145\/2576768.2598286"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/j.infsof.2016.01.016","article-title":"Subtle higher order mutants","volume":"81","author":"Omar","year":"2017","journal-title":"Inf. Softw. Technol."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1173","DOI":"10.3233\/JIFS-169117","article-title":"Addressing mutation testing problems by applying multi-objective optimization algorithms and higher order mutation","volume":"32","author":"Nguyen","year":"2017","journal-title":"J. Intell. Fuzzy Syst."},{"key":"ref_39","first-page":"34","article-title":"An Approach for the Generation of Higher Order Mutants Using Genetic Algorithms","volume":"10","author":"Abuljadayel","year":"2018","journal-title":"Int. J. Intell. Syst. Appl."},{"key":"ref_40","unstructured":"Nguyen, Q.V. (December, January 30). Increasing Mutation Testing Effectiveness by Combining Lower Order Mutants to Construct Higher Order Mutants. Proceedings of the International Conference on Computational Collective Intelligence, Da Nang, Vietnam."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Jang, J.I., Ryu, D., and Baik, J. (2021). HOTFUZ: Cost-effective higher-order mutation-based fault localization. Softw. Test. Verif. Reliab., e1802.","DOI":"10.1002\/stvr.1802"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1157","DOI":"10.1109\/TR.2022.3162039","article-title":"Can Higher-Order Mutants Improve the Performance of Mutation-Based Fault Localization?","volume":"71","author":"Wang","year":"2022","journal-title":"IEEE Trans. Reliab."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1137\/0202009","article-title":"Genetic algorithms and the optimal allocation of trials","volume":"2","author":"Holland","year":"1973","journal-title":"SIAM J. Comput."},{"key":"ref_44","unstructured":"Goldberg, D.E. (1989). Genetic Algorithms in Search, Optimization and Machine Learning, Addison-Wesley Longman Publishing Co., Inc.. [1st ed.]."},{"key":"ref_45","unstructured":"Baker, J.E. (1987, January 28\u201331). Reducing Bias and Inefficiency in the Selection Algorithm. Proceedings of the Second International Conference on Genetic Algorithms on Genetic Algorithms and Their Application, Cambridge, MA, USA."},{"key":"ref_46","first-page":"193","article-title":"Genetic algorithms, tournament selection and the effects of noise","volume":"9","author":"Miller","year":"1995","journal-title":"Complex Syst."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"396","DOI":"10.1073\/pnas.76.1.396","article-title":"Efficiency of truncation selection","volume":"76","author":"Crow","year":"1979","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_48","unstructured":"Baker, J.E. (1995, January 12\u201314). Adaptive Selection Methods for Genetic Algorithms. Proceedings of the 1st International Conference on Genetic Algorithms, Sheffield, UK."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1007\/BF00175354","article-title":"A genetic algorithm tutorial","volume":"4","author":"Whitley","year":"1994","journal-title":"Stat. Comput."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"1155","DOI":"10.1002\/spe.2346","article-title":"Spoon: A Library for Implementing Analyses and Transformations of Java Source Code","volume":"46","author":"Pawlak","year":"2015","journal-title":"Softw. Pract. Exp."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Pacheco, C., and Ernst, M.D. (2007, January 21\u201325). Randoop: Feedback-directed random testing for Java. Proceedings of the Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion, Montreal, QC, Canada.","DOI":"10.1145\/1297846.1297902"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Fraser, G., and Andrea, A. (2011, January 5\u20139). Evosuite: Automatic test suite generation for object-oriented software. Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, Szeged, Hungary.","DOI":"10.1145\/2025113.2025179"},{"key":"ref_53","unstructured":"Kerlinger, F.N. (1966). Foundations of Behavioral Research, Rinehart and Winston."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/7\/327\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:43:52Z","timestamp":1760139832000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/7\/327"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,7]]},"references-count":53,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2022,7]]}},"alternative-id":["info13070327"],"URL":"https:\/\/doi.org\/10.3390\/info13070327","relation":{},"ISSN":["2078-2489"],"issn-type":[{"type":"electronic","value":"2078-2489"}],"subject":[],"published":{"date-parts":[[2022,7,7]]}}}