{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,22]],"date-time":"2026-07-22T04:48:58Z","timestamp":1784695738347,"version":"3.55.0"},"publisher-location":"New York, NY, USA","reference-count":45,"publisher":"ACM","license":[{"start":{"date-parts":[[2023,7,12]],"date-time":"2023-07-12T00:00:00Z","timestamp":1689120000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,7,15]]},"DOI":"10.1145\/3583131.3590379","type":"proceedings-article","created":{"date-parts":[[2023,7,12]],"date-time":"2023-07-12T19:40:19Z","timestamp":1689190819000},"page":"1490-1498","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["Searching for Quality: Genetic Algorithms and Metamorphic Testing for Software Engineering ML"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4341-8840","authenticated-orcid":false,"given":"Leonhard","family":"Applis","sequence":"first","affiliation":[{"name":"Delft University of Technology, Delft, Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7395-3588","authenticated-orcid":false,"given":"Annibale","family":"Panichella","sequence":"additional","affiliation":[{"name":"Delft University of Technology, Delft, Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-9886-6724","authenticated-orcid":false,"given":"Ruben","family":"Marang","sequence":"additional","affiliation":[{"name":"Delft University of Technology, Delft, Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,7,12]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Fairness testing: A comprehensive survey and analysis of trends,\" arXiv preprint arXiv:2207.10223","author":"Chen Z.","year":"2022","unstructured":"Z. Chen , J. M. Zhang , M. Hort , F. Sarro , and M. Harman , \" Fairness testing: A comprehensive survey and analysis of trends,\" arXiv preprint arXiv:2207.10223 , 2022 . Z. Chen, J. M. Zhang, M. Hort, F. Sarro, and M. Harman, \"Fairness testing: A comprehensive survey and analysis of trends,\" arXiv preprint arXiv:2207.10223, 2022."},{"key":"e_1_3_2_1_2_1","volume-title":"Diagnostics and extrapolation in machine learning. stanford university","author":"Hooker G.","year":"2004","unstructured":"G. Hooker , Diagnostics and extrapolation in machine learning. stanford university , 2004 . G. Hooker, Diagnostics and extrapolation in machine learning. stanford university, 2004."},{"key":"e_1_3_2_1_3_1","first-page":"129","volume-title":"An empirical study on tensorflow program bugs,\" in Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis","author":"Zhang Y.","year":"2018","unstructured":"Y. Zhang , Y. Chen , S.-C. Cheung , Y. Xiong , and L. Zhang , \" An empirical study on tensorflow program bugs,\" in Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis , 2018 , pp. 129 -- 140 . Y. Zhang, Y. Chen, S.-C. Cheung, Y. Xiong, and L. Zhang, \"An empirical study on tensorflow program bugs,\" in Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2018, pp. 129--140."},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2962027"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-018-05707-8"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3428230"},{"key":"e_1_3_2_1_7_1","first-page":"243","volume-title":"Embedding java classes with code2vec: Improvements from variable obfuscation,\" in Proceedings of the 17th International Conference on Mining Software Repositories","author":"Compton R.","year":"2020","unstructured":"R. Compton , E. Frank , P. Patros , and A. Koay , \" Embedding java classes with code2vec: Improvements from variable obfuscation,\" in Proceedings of the 17th International Conference on Mining Software Repositories , 2020 , pp. 243 -- 253 . R. Compton, E. Frank, P. Patros, and A. Koay, \"Embedding java classes with code2vec: Improvements from variable obfuscation,\" in Proceedings of the 17th International Conference on Mining Software Repositories, 2020, pp. 243--253."},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2021.106552"},{"key":"e_1_3_2_1_9_1","first-page":"1377","volume-title":"IEEE","author":"Applis L.","year":"2021","unstructured":"L. Applis , A. Panichella , and A. van Deursen , \"Assessing robustness of ml-based program analysis tools using metamorphic program transformations,\" in 2021 36th IEEE\/ACM International Conference on Automated Software Engineering (ASE) . IEEE , 2021 , pp. 1377 -- 1381 . L. Applis, A. Panichella, and A. van Deursen, \"Assessing robustness of ml-based program analysis tools using metamorphic program transformations,\" in 2021 36th IEEE\/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2021, pp. 1377--1381."},{"key":"e_1_3_2_1_10_1","first-page":"1","volume-title":"IEEE","author":"Ahmad K.","year":"2021","unstructured":"K. Ahmad , M. Bano , M. Abdelrazek , C. Arora , and J. Grundy , \" What's up with requirements engineering for artificial intelligence systems?\" in 2021 IEEE 29th International Requirements Engineering Conference (RE) . IEEE , 2021 , pp. 1 -- 12 . K. Ahmad, M. Bano, M. Abdelrazek, C. Arora, and J. Grundy, \"What's up with requirements engineering for artificial intelligence systems?\" in 2021 IEEE 29th International Requirements Engineering Conference (RE). IEEE, 2021, pp. 1--12."},{"key":"e_1_3_2_1_11_1","first-page":"1","article-title":"\"code2vec: Learning distributed representations of code,\" Proceedings of the ACM on Programming Languages, vol. 3, no","author":"Alon U.","year":"2019","unstructured":"U. Alon , M. Zilberstein , O. Levy , and E. Yahav , \"code2vec: Learning distributed representations of code,\" Proceedings of the ACM on Programming Languages, vol. 3, no . POPL , pp. 1 -- 29 , 2019 . U. Alon, M. Zilberstein, O. Levy, and E. Yahav, \"code2vec: Learning distributed representations of code,\" Proceedings of the ACM on Programming Languages, vol. 3, no. POPL, pp. 1--29, 2019.","journal-title":"POPL"},{"key":"e_1_3_2_1_12_1","first-page":"548","volume-title":"Machine learning for software engineering: Models, methods, and applications,\" in 2018 IEEE\/ACM 40th International Conference on Software Engineering: Companion (ICSE-Companion)","author":"Meinke K.","year":"2018","unstructured":"K. Meinke and A. Bennaceur , \" Machine learning for software engineering: Models, methods, and applications,\" in 2018 IEEE\/ACM 40th International Conference on Software Engineering: Companion (ICSE-Companion) , 2018 , pp. 548 -- 549 . K. Meinke and A. Bennaceur, \"Machine learning for software engineering: Models, methods, and applications,\" in 2018 IEEE\/ACM 40th International Conference on Software Engineering: Companion (ICSE-Companion), 2018, pp. 548--549."},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2532875"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2010.11.920"},{"key":"e_1_3_2_1_15_1","first-page":"867","volume-title":"SEKE 2008","author":"Murphy C.","year":"2008","unstructured":"C. Murphy , G. Kaiser , L. Hu , and L. Wu , \" Properties of machine learning applications for use in metamorphic testing,\" 20th International Conference on Software Engineering and Knowledge Engineering , SEKE 2008 , pp. 867 -- 872 , 2008 . C. Murphy, G. Kaiser, L. Hu, and L. Wu, \"Properties of machine learning applications for use in metamorphic testing,\" 20th International Conference on Software Engineering and Knowledge Engineering, SEKE 2008, pp. 867--872, 2008."},{"key":"e_1_3_2_1_16_1","first-page":"125","article-title":"Counterfactual explanations for models of code,\" in Proceedings of the 44th International Conference on Software Engineering","author":"Cito J.","year":"2022","unstructured":"J. Cito , I. Dillig , V. Murali , and S. Chandra , \" Counterfactual explanations for models of code,\" in Proceedings of the 44th International Conference on Software Engineering : Software Engineering in Practice , 2022 , pp. 125 -- 134 . J. Cito, I. Dillig, V. Murali, and S. Chandra, \"Counterfactual explanations for models of code,\" in Proceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice, 2022, pp. 125--134.","journal-title":"Software Engineering in Practice"},{"key":"e_1_3_2_1_17_1","volume-title":"Jiang et al., \"Codebert: A pre-trained model for programming and natural languages,\" arXiv preprint arXiv:2002.08155","author":"Feng Z.","year":"2020","unstructured":"Z. Feng , D. Guo , D. Tang , N. Duan , X. Feng , M. Gong , L. Shou , B. Qin , T. Liu , D. Jiang et al., \"Codebert: A pre-trained model for programming and natural languages,\" arXiv preprint arXiv:2002.08155 , 2020 . Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang et al., \"Codebert: A pre-trained model for programming and natural languages,\" arXiv preprint arXiv:2002.08155, 2020."},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.294"},{"key":"e_1_3_2_1_19_1","first-page":"416","volume-title":"Evosuite: automatic test suite generation for object-oriented software,\" in Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering","author":"Fraser G.","year":"2011","unstructured":"G. Fraser and A. Arcuri , \" Evosuite: automatic test suite generation for object-oriented software,\" in Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering , 2011 , pp. 416 -- 419 . G. Fraser and A. Arcuri, \"Evosuite: automatic test suite generation for object-oriented software,\" in Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering, 2011, pp. 416--419."},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1013886.1007528"},{"key":"e_1_3_2_1_21_1","volume-title":"Regression testing minimization, selection and prioritization: a survey,\" Software testing, verification and reliability","author":"Yoo S.","unstructured":"S. Yoo and M. Harman , \" Regression testing minimization, selection and prioritization: a survey,\" Software testing, verification and reliability , vol. 22 , no. 2, pp. 67--120, 2012. S. Yoo and M. Harman, \"Regression testing minimization, selection and prioritization: a survey,\" Software testing, verification and reliability, vol. 22, no. 2, pp. 67--120, 2012."},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"crossref","unstructured":"S. Yoo \"Pareto efficient multi-objective test case selection \" in Proceedings of the 2007 international symposium on Software testing and analysis 2007 pp. 140--150.  S. Yoo \"Pareto efficient multi-objective test case selection \" in Proceedings of the 2007 international symposium on Software testing and analysis 2007 pp. 140--150.","DOI":"10.1145\/1273463.1273483"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2016.01.017"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-006-7551-5"},{"key":"e_1_3_2_1_25_1","volume-title":"Genprog: A generic method for automatic software repair,\" Ieee transactions on software engineering","author":"Goues C. Le","unstructured":"C. Le Goues , T. Nguyen , S. Forrest , and W. Weimer , \" Genprog: A generic method for automatic software repair,\" Ieee transactions on software engineering , vol. 38 , no. 1, pp. 54--72, 2011. C. Le Goues, T. Nguyen, S. Forrest, and W. Weimer, \"Genprog: A generic method for automatic software repair,\" Ieee transactions on software engineering, vol. 38, no. 1, pp. 54--72, 2011."},{"key":"e_1_3_2_1_26_1","first-page":"441","volume-title":"Astor: A program repair library for java,\" in Proceedings of the 25th International Symposium on Software Testing and Analysis","author":"Martinez M.","year":"2016","unstructured":"M. Martinez and M. Monperrus , \" Astor: A program repair library for java,\" in Proceedings of the 25th International Symposium on Software Testing and Analysis , 2016 , pp. 441 -- 444 . M. Martinez and M. Monperrus, \"Astor: A program repair library for java,\" in Proceedings of the 25th International Symposium on Software Testing and Analysis, 2016, pp. 441--444."},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2018.08.010"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2018.08.009"},{"key":"e_1_3_2_1_29_1","volume-title":"Single and multi-objective test cases prioritization for self-driving cars in virtual environments,\" Proceedings of the ACM on Measurement and Analysis of Computing Systems","author":"Birchler C.","year":"2022","unstructured":"C. Birchler , S. Khatiri , P. Derakhshanfar , S. Panichella , and A. Panichella , \" Single and multi-objective test cases prioritization for self-driving cars in virtual environments,\" Proceedings of the ACM on Measurement and Analysis of Computing Systems , 2022 . C. Birchler, S. Khatiri, P. Derakhshanfar, S. Panichella, and A. Panichella, \"Single and multi-objective test cases prioritization for self-driving cars in virtual environments,\" Proceedings of the ACM on Measurement and Analysis of Computing Systems, 2022."},{"key":"e_1_3_2_1_30_1","first-page":"20","volume-title":"IEEE","author":"Panichella S.","year":"2021","unstructured":"S. Panichella , A. Gambi , F. Zampetti , and V. Riccio , \" Sbst tool competition 2021,\" in 2021 IEEE\/ACM 14th International Workshop on Search-Based Software Testing (SBST) . IEEE , 2021 , pp. 20 -- 27 . S. Panichella, A. Gambi, F. Zampetti, and V. Riccio, \"Sbst tool competition 2021,\" in 2021 IEEE\/ACM 14th International Workshop on Search-Based Software Testing (SBST). IEEE, 2021, pp. 20--27."},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/1090.001.0001"},{"key":"e_1_3_2_1_32_1","volume-title":"Genetic programming: principles and applications,\" Engineering applications of artificial intelligence","author":"Sette S.","unstructured":"S. Sette and L. Boullart , \" Genetic programming: principles and applications,\" Engineering applications of artificial intelligence , vol. 14 , no. 6, pp. 727--736, 2001. S. Sette and L. Boullart, \"Genetic programming: principles and applications,\" Engineering applications of artificial intelligence, vol. 14, no. 6, pp. 727--736, 2001."},{"key":"e_1_3_2_1_33_1","first-page":"1","volume-title":"IEEE","author":"Panichella A.","year":"2015","unstructured":"A. Panichella , F. M. Kifetew , and P. Tonella , \" Reformulating branch coverage as a many-objective optimization problem,\" in 2015 IEEE 8th international conference on software testing, verification and validation (ICST) . IEEE , 2015 , pp. 1 -- 10 . A. Panichella, F. M. Kifetew, and P. Tonella, \"Reformulating branch coverage as a many-objective optimization problem,\" in 2015 IEEE 8th international conference on software testing, verification and validation (ICST). IEEE, 2015, pp. 1--10."},{"key":"e_1_3_2_1_34_1","first-page":"362","volume-title":"IEEE","author":"Fraser G.","year":"2013","unstructured":"G. Fraser and A. Arcuri , \" Evosuite: On the challenges of test case generation in the real world,\" in 2013 IEEE sixth international conference on software testing, verification and validation . IEEE , 2013 , pp. 362 -- 369 . G. Fraser and A. Arcuri, \"Evosuite: On the challenges of test case generation in the real world,\" in 2013 IEEE sixth international conference on software testing, verification and validation. IEEE, 2013, pp. 362--369."},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2663435"},{"key":"e_1_3_2_1_36_1","first-page":"254","volume-title":"The strength of random search on automated program repair,\" in Proceedings of the 36th International Conference on Software Engineering","author":"Qi Y.","year":"2014","unstructured":"Y. Qi , X. Mao , Y. Lei , Z. Dai , and C. Wang , \" The strength of random search on automated program repair,\" in Proceedings of the 36th International Conference on Software Engineering , 2014 , pp. 254 -- 265 . Y. Qi, X. Mao, Y. Lei, Z. Dai, and C. Wang, \"The strength of random search on automated program repair,\" in Proceedings of the 36th International Conference on Software Engineering, 2014, pp. 254--265."},{"issue":"2","key":"e_1_3_2_1_37_1","article-title":"Random search for hyper-parameter optimization","volume":"13","author":"Bergstra J.","year":"2012","unstructured":"J. Bergstra and Y. Bengio , \" Random search for hyper-parameter optimization .\" Journal of machine learning research , vol. 13 , no. 2 , 2012 . J. Bergstra and Y. Bengio, \"Random search for hyper-parameter optimization.\" Journal of machine learning research, vol. 13, no. 2, 2012.","journal-title":"Journal of machine learning research"},{"key":"e_1_3_2_1_38_1","first-page":"1","volume-title":"A practical guide for using statistical tests to assess randomized algorithms in software engineering,\" in Proceedings of the 33rd international conference on software engineering","author":"Arcuri A.","year":"2011","unstructured":"A. Arcuri and L. Briand , \" A practical guide for using statistical tests to assess randomized algorithms in software engineering,\" in Proceedings of the 33rd international conference on software engineering , 2011 , pp. 1 -- 10 . A. Arcuri and L. Briand, \"A practical guide for using statistical tests to assess randomized algorithms in software engineering,\" in Proceedings of the 33rd international conference on software engineering, 2011, pp. 1--10."},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1486"},{"key":"e_1_3_2_1_40_1","volume-title":"Practical nonparametric statistics. john wiley & sons","author":"Conover W. J.","year":"1999","unstructured":"W. J. Conover , Practical nonparametric statistics. john wiley & sons , 1999 , vol. 350 . W. J. Conover, Practical nonparametric statistics. john wiley & sons, 1999, vol. 350."},{"issue":"2","key":"e_1_3_2_1_41_1","first-page":"101","article-title":"A critique and improvement of the CL common language effect size statistics of McGraw and Wong","volume":"25","author":"Vargha A.","year":"2000","unstructured":"A. Vargha and H. D. Delaney , \" A critique and improvement of the CL common language effect size statistics of McGraw and Wong ,\" Journal of Educational and Behavioral Statistics , vol. 25 , no. 2 , pp. 101 -- 132 , 2000 . A. Vargha and H. D. Delaney, \"A critique and improvement of the CL common language effect size statistics of McGraw and Wong,\" Journal of Educational and Behavioral Statistics, vol. 25, no. 2, pp. 101--132, 2000.","journal-title":"Journal of Educational and Behavioral Statistics"},{"key":"e_1_3_2_1_42_1","first-page":"623","volume-title":"A comparison of statistical significance tests for information retrieval evaluation,\" in Proceedings of the sixteenth ACM conference on Conference on information and knowledge management","author":"Smucker M. D.","year":"2007","unstructured":"M. D. Smucker , J. Allan , and B. Carterette , \" A comparison of statistical significance tests for information retrieval evaluation,\" in Proceedings of the sixteenth ACM conference on Conference on information and knowledge management , 2007 , pp. 623 -- 632 . M. D. Smucker, J. Allan, and B. Carterette, \"A comparison of statistical significance tests for information retrieval evaluation,\" in Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, 2007, pp. 623--632."},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.2307\/2333709"},{"key":"e_1_3_2_1_44_1","first-page":"88","volume-title":"ISSTA 2020","author":"Abdessalem R. B.","year":"2020","unstructured":"R. B. Abdessalem , A. Panichella , S. Nejati , L. C. Briand , and T. Stifter , \" Automated repair of feature interaction failures in automated driving systems,\" in Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, ser . ISSTA 2020 . New York, NY, USA: Association for Computing Machinery , 2020 , p. 88 -- 100 . R. B. Abdessalem, A. Panichella, S. Nejati, L. C. Briand, and T. Stifter, \"Automated repair of feature interaction failures in automated driving systems,\" in Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2020. New York, NY, USA: Association for Computing Machinery, 2020, p. 88--100."},{"key":"e_1_3_2_1_45_1","volume-title":"A survey on handling computationally expensive multiobjective optimization problems with evolutionary algorithms,\" Soft Computing","author":"Chugh T.","year":"2019","unstructured":"T. Chugh , K. Sindhya , J. Hakanen , and K. Miettinen , \" A survey on handling computationally expensive multiobjective optimization problems with evolutionary algorithms,\" Soft Computing , vol. 23 , 2019 . T. Chugh, K. Sindhya, J. Hakanen, and K. Miettinen, \"A survey on handling computationally expensive multiobjective optimization problems with evolutionary algorithms,\" Soft Computing, vol. 23, 2019."}],"event":{"name":"GECCO '23: Genetic and Evolutionary Computation Conference","location":"Lisbon Portugal","acronym":"GECCO '23","sponsor":["SIGEVO ACM Special Interest Group on Genetic and Evolutionary Computation"]},"container-title":["Proceedings of the Genetic and Evolutionary Computation Conference"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3583131.3590379","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3583131.3590379","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:48:49Z","timestamp":1750182529000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3583131.3590379"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,12]]},"references-count":45,"alternative-id":["10.1145\/3583131.3590379","10.1145\/3583131"],"URL":"https:\/\/doi.org\/10.1145\/3583131.3590379","relation":{},"subject":[],"published":{"date-parts":[[2023,7,12]]},"assertion":[{"value":"2023-07-12","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}