{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,20]],"date-time":"2026-02-20T23:21:33Z","timestamp":1771629693736,"version":"3.50.1"},"reference-count":63,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,11,16]],"date-time":"2024-11-16T00:00:00Z","timestamp":1731715200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,11,16]],"date-time":"2024-11-16T00:00:00Z","timestamp":1731715200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100018707","name":"HORIZON EUROPE Reforming and enhancing the European Research and Innovation system","doi-asserted-by":"publisher","award":["956123"],"award-info":[{"award-number":["956123"]}],"id":[{"id":"10.13039\/100018707","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100005713","name":"Technische Universit\u00e4t M\u00fcnchen","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100005713","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2025,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Search-based software testing (SBST) is a widely-adopted technique for testing complex systems with large input spaces, such as Deep Learning-enabled (DL-enabled) systems. Many SBST techniques focus on Pareto-based optimization where multiple objectives are optimized in parallel to reveal failures. However, it is important to ensure that identified failures are spread throughout the entire failure-inducing area of a search domain, and not clustered in a sub-region. This ensures that identified failures are semantically diverse and reveal a wide range of underlying causes. In this paper, we present a theoretical argument explaining why testing based on Pareto optimization is inadequate for covering failure-inducing areas within a search domain. We support our argument with empirical results obtained by applying two widely used types of Pareto-based optimization techniques, namely NSGA-II (an evolutionary algorithm) and OMOPSO (a swarm-based algorithm), to two DL-enabled systems: an industrial Automated Valet Parking (AVP) system and a system for classifying handwritten digits. We measure the coverage of failure-revealing test inputs in the input space using a metric, that we refer to as the <jats:italic>Coverage Inverted Distance<\/jats:italic> (CID) quality indicator. Our results show that NSGA-II and OMOPSO are <jats:italic>not<\/jats:italic> more effective than a na\u00efve random search baseline in covering test inputs that reveal failures. We show that this comparison remains valid for failure-inducing regions of various sizes of these two case studies. Further, we show that incorporating a diversity-focused fitness function as well as a repopulation operator in NSGA-II improves, on average, the coverage difference between NSGA-II and random search by 52.1%. However, even after diversification, NSGA-II still does not outperform random testing in covering test inputs that reveal failures. The replication package for this study is available in a GitHub repository\u00a0(Replication package. <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/ast-fortiss-tum\/coverage-emse-24\">https:\/\/github.com\/ast-fortiss-tum\/coverage-emse-24<\/jats:ext-link> 2024.<\/jats:p>","DOI":"10.1007\/s10664-024-10564-3","type":"journal-article","created":{"date-parts":[[2024,11,16]],"date-time":"2024-11-16T01:23:24Z","timestamp":1731720204000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Can search-based testing with pareto optimization effectively cover failure-revealing test inputs?"],"prefix":"10.1007","volume":"30","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-1162-6252","authenticated-orcid":false,"given":"Lev","family":"Sorokin","sequence":"first","affiliation":[]},{"given":"Damir","family":"Safin","sequence":"additional","affiliation":[]},{"given":"Shiva","family":"Nejati","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,11,16]]},"reference":[{"key":"10564_CR1","unstructured":"(2024) Replication package. https:\/\/github.com\/ast-fortiss-tum\/coverage-emse-24"},{"key":"10564_CR2","doi-asserted-by":"publisher","unstructured":"Aghababaeyan Z, Abdellatif M, Briand L, S R, Bagherzadeh M (2023) Black-box testing of deep neural networks through test case diversity. IEEE Trans Softw pp 3182\u20133204, https:\/\/doi.org\/10.1109\/TSE.2023.3243522","DOI":"10.1109\/TSE.2023.3243522"},{"key":"10564_CR3","doi-asserted-by":"crossref","unstructured":"Arcuri A, Fraser G (2011) On parameter tuning in search based software engineering. SSBSE, p 33\u201347","DOI":"10.1007\/978-3-642-23716-4_6"},{"key":"10564_CR4","doi-asserted-by":"publisher","unstructured":"Ben\u00a0Abdessalem R, Nejati S, Briand LC, Stifter T (2016) Testing advanced driver assistance systems using multi-objective search and neural networks. ASE, p 63\u201374, https:\/\/doi.org\/10.1145\/2970276.2970311","DOI":"10.1145\/2970276.2970311"},{"key":"10564_CR5","doi-asserted-by":"publisher","unstructured":"Ben\u00a0Abdessalem R, Nejati S, C\u00a0Briand L, Stifter T (2018) Testing vision-based control systems using learnable evolutionary algorithms. In: ICSE, pp 1016\u20131026, https:\/\/doi.org\/10.1145\/3180155.3180160","DOI":"10.1145\/3180155.3180160"},{"key":"10564_CR6","doi-asserted-by":"crossref","unstructured":"Biagiola M, Stocco A, Ricca F, Tonella P (2019) Diversity-based web test generation. ESEC\/FSE, p 142\u2013153","DOI":"10.1145\/3338906.3338970"},{"issue":"2","key":"10564_CR7","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3533818","volume":"32","author":"C Birchler","year":"2023","unstructured":"Birchler C, Khatiri S, Derakhshanfar P, Panichella S, Panichella A (2023) Single and multi-objective test cases prioritization for self-driving cars in virtual environments. ACM Trans Softw Eng Methodol 32(2):1\u201330. https:\/\/doi.org\/10.1145\/3533818","journal-title":"ACM Trans Softw Eng Methodol"},{"key":"10564_CR8","doi-asserted-by":"crossref","unstructured":"Borg M, Abdessalem RB, Nejati S, Jegeden F, Shin D (2021) Digital twins are not monozygotic - cross-replicating ADAS testing in two industry-grade automotive simulators. In: 14th IEEE Conference on Software Testing, Verification and Validation (ICST), pp 383\u2013393","DOI":"10.1109\/ICST49551.2021.00050"},{"key":"10564_CR9","unstructured":"Bosch (2023) Automated valet parking. https:\/\/www.bosch-mobility.com\/en\/solutions\/parking\/automated-valet-parking\/"},{"key":"10564_CR10","doi-asserted-by":"publisher","unstructured":"Bridson R (2007) Fast poisson disk sampling in arbitrary dimensions. SIGGRAPH \u201907, p 22\u2013es, https:\/\/doi.org\/10.1145\/1278780.1278807","DOI":"10.1145\/1278780.1278807"},{"key":"10564_CR11","doi-asserted-by":"crossref","unstructured":"Browne C, Powley EJ, Whitehouse D, Lucas SMM, Cowling PI, Rohlfshagen P, Tavener S, Liebana DP, Samothrakis S, Colton S (2012) A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in Games 4:1\u201343, https:\/\/api.semanticscholar.org\/CorpusID:9316331","DOI":"10.1109\/TCIAIG.2012.2186810"},{"key":"10564_CR12","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pcbi.1000187","volume":"9","author":"J Clune","year":"2008","unstructured":"Clune J, Misevic D, Ofria C, Lenski RE, Elena SF, Sanju\u00e1n R (2008) Natural selection fails to optimize mutation rates for long-term adaptation on rugged fitness landscapes. PLoS Comput Biol 9:1\u20138. https:\/\/doi.org\/10.1371\/journal.pcbi.1000187","journal-title":"PLoS Comput Biol"},{"key":"10564_CR13","doi-asserted-by":"publisher","unstructured":"Clune J, Goings S, Punch B, Goodman E (2005) Investigations in meta-gas: panaceas or pipe dreams? In: Proceedings of the 7th annual workshop on genetic and evolutionary computation, Association for Computing Machinery, New York, NY, USA, GECCO \u201905, p 235\u2013241, https:\/\/doi.org\/10.1145\/1102256.1102311","DOI":"10.1145\/1102256.1102311"},{"key":"10564_CR14","doi-asserted-by":"crossref","unstructured":"Coello CAC, Pulido GT, Lechuga MS (2004) Handling multiple objectives with particle swarm optimization. IEEE Transactions on Evolutionary Computation 8:256\u2013279. https:\/\/api.semanticscholar.org\/CorpusID:10783227","DOI":"10.1109\/TEVC.2004.826067"},{"issue":"3","key":"10564_CR15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2480741.2480752","volume":"45","author":"M \u010crepin\u0161ek","year":"2013","unstructured":"\u010crepin\u0161ek M, Liu SH, Mernik M (2013) Exploration and exploitation in evolutionary algorithms: a survey. ACM Comput Surv 45(3):1\u201333. https:\/\/doi.org\/10.1145\/2480741.2480752","journal-title":"ACM Comput Surv"},{"issue":"2","key":"10564_CR16","doi-asserted-by":"publisher","first-page":"182","DOI":"10.1109\/4235.996017","volume":"6","author":"K Deb","year":"2002","unstructured":"Deb K, Pratap A, Agarwal S, Meyarivan T (2002) A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE Trans Evol Comput 6(2):182\u2013197. https:\/\/doi.org\/10.1109\/4235.996017","journal-title":"IEEE Trans Evol Comput"},{"key":"10564_CR17","doi-asserted-by":"publisher","first-page":"394","DOI":"10.1017\/jpr.2017.7","volume":"54","author":"L Devroye","year":"2015","unstructured":"Devroye L, Gy\u00f6rfi L, Lugosi G, Walk H (2015) On the measure of voronoi cells. J Appl Probab 54:394\u2013408. https:\/\/doi.org\/10.1017\/jpr.2017.7","journal-title":"J Appl Probab"},{"key":"10564_CR18","doi-asserted-by":"publisher","unstructured":"Ebadi H, Moghadam M, Borg M, Gay G, Fontes A, Socha K (2021) Efficient and effective generation of test cases for pedestrian detection - search-based software testing of baidu apollo in svl. In: 2021 IEEE International conference on artificial intelligence testing (AITest), IEEE Computer Society, Los Alamitos, CA, USA, pp 103\u2013110, https:\/\/doi.org\/10.1109\/AITEST52744.2021.00030","DOI":"10.1109\/AITEST52744.2021.00030"},{"key":"10564_CR19","doi-asserted-by":"publisher","first-page":"1305","DOI":"10.1109\/83.623193","volume":"6","author":"Y Eldar","year":"1997","unstructured":"Eldar Y, Lindenbaum M, Porat M, Zeevi Y (1997) The farthest point strategy for progressive image sampling. IEEE Trans Image Process 6:1305\u201315. https:\/\/doi.org\/10.1109\/83.623193","journal-title":"IEEE Trans Image Process"},{"key":"10564_CR20","unstructured":"Feldt R, Poulding S (2017) Searching for test data with feature diversity. arXiv:1709.06017"},{"key":"10564_CR21","doi-asserted-by":"publisher","unstructured":"Feldt R, Poulding S, Clark D, Yoo S (2016) Test set diameter: quantifying the diversity of sets of test cases. In: ICST, pp 223\u2013233, https:\/\/doi.org\/10.1109\/ICST.2016.33","DOI":"10.1109\/ICST.2016.33"},{"issue":"3","key":"10564_CR22","doi-asserted-by":"publisher","first-page":"1991","DOI":"10.1007\/s12065-021-00607-9","volume":"15","author":"P Fuangkhon","year":"2022","unstructured":"Fuangkhon P (2022) Effect of the distance functions on the distance-based instance selection for the feed-forward neural network. Evol Intel 15(3):1991\u20132015. https:\/\/doi.org\/10.1007\/s12065-021-00607-9","journal-title":"Evol Intel"},{"issue":"3","key":"10564_CR23","doi-asserted-by":"publisher","first-page":"958","DOI":"10.1214\/aoms\/1177704464","volume":"33","author":"EN Gilbert","year":"1962","unstructured":"Gilbert EN (1962) Random subdivisions of space into crystals. Ann Math Stat 33(3):958\u2013972. https:\/\/doi.org\/10.1214\/aoms\/1177704464","journal-title":"Ann Math Stat"},{"key":"10564_CR24","doi-asserted-by":"publisher","unstructured":"Humeniuk D, Khomh F, Antoniol G (2022) A search-based framework for automatic generation of testing environments for cyber\u2013physical systems. Inf Softw Technol 149:106936. https:\/\/doi.org\/10.1016\/j.infsof.2022.106936, https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0950584922000866","DOI":"10.1016\/j.infsof.2022.106936"},{"key":"10564_CR25","doi-asserted-by":"publisher","unstructured":"Humeniuk D, Khomh F, Antoniol G (2023) Ambiegen: a search-based framework for autonomous systems testingimage 1. Sci Comput Program 230:102990. https:\/\/doi.org\/10.1016\/j.scico.2023.102990, https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0167642323000722","DOI":"10.1016\/j.scico.2023.102990"},{"key":"10564_CR26","doi-asserted-by":"publisher","DOI":"10.1145\/3680468","author":"D Humeniuk","year":"2024","unstructured":"Humeniuk D, Khomh F, Antoniol G (2024) Reinforcement learning informed evolutionary search for autonomous systems testing. ACM Trans Softw Eng Methodol. https:\/\/doi.org\/10.1145\/3680468","journal-title":"ACM Trans Softw Eng Methodol"},{"key":"10564_CR27","doi-asserted-by":"crossref","unstructured":"Hungar H (2020) A concept of scenario space exploration with criticality coverage guarantees - extended abstract. ISOLA, pp 293\u2013306, https:\/\/elib.dlr.de\/137353\/","DOI":"10.1007\/978-3-030-61467-6_19"},{"key":"10564_CR28","doi-asserted-by":"publisher","unstructured":"Jahangirova G, Stocco A, Tonella P (2021) Quality metrics and oracles for autonomous vehicles testing. In: 2021 14th IEEE Conference on Software Testing, verification and validation (ICST), pp 194\u2013204, https:\/\/doi.org\/10.1109\/ICST49551.2021.00030","DOI":"10.1109\/ICST49551.2021.00030"},{"key":"10564_CR29","doi-asserted-by":"publisher","unstructured":"Jodat BA, Chandar A, Nejati S, Sabetzadeh M (2024) Test generation strategies for building failure models and explaining spurious failures. ACM Trans Softw Eng Methodol 33(4), https:\/\/doi.org\/10.1145\/3638246","DOI":"10.1145\/3638246"},{"key":"10564_CR30","doi-asserted-by":"publisher","unstructured":"Kl\u00fcck F, Zimmermann M, Wotawa F, Nica M (2019) Genetic algorithm-based test parameter optimization for adas system testing. In: 2019 IEEE 19th International conference on software Quality, Reliability and Security (QRS), pp 418\u2013425, https:\/\/doi.org\/10.1109\/QRS.2019.00058","DOI":"10.1109\/QRS.2019.00058"},{"key":"10564_CR31","doi-asserted-by":"publisher","unstructured":"Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278\u20132324. https:\/\/doi.org\/10.1109\/5.726791","DOI":"10.1109\/5.726791"},{"key":"10564_CR32","unstructured":"LeCun Y, Cortes C (2005) The mnist database of handwritten digits. https:\/\/api.semanticscholar.org\/CorpusID:60282629"},{"key":"10564_CR33","doi-asserted-by":"publisher","unstructured":"Lehman J, Stanley KO (2011a) Abandoning objectives: evolution through the search for novelty alone. Evol Comput 19(2):189\u2013223, https:\/\/doi.org\/10.1162\/EVCO_a_00025","DOI":"10.1162\/EVCO_a_00025"},{"key":"10564_CR34","doi-asserted-by":"publisher","unstructured":"Lehman J, Stanley KO (2011b) Evolving a diversity of virtual creatures through novelty search and local competition. In: Proceedings of the 13th annual conference on genetic and evolutionary computation, Association for Computing Machinery, New York, NY, USA, GECCO \u201911, p 211\u2013218. https:\/\/doi.org\/10.1145\/2001576.2001606","DOI":"10.1145\/2001576.2001606"},{"issue":"2","key":"10564_CR35","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3300148","volume":"52","author":"M Li","year":"2019","unstructured":"Li M, Yao X (2019) Quality evaluation of solution sets in multiobjective optimisation: A survey. ACM Comput Surv 52(2):1\u201338. https:\/\/doi.org\/10.1145\/3300148","journal-title":"ACM Comput Surv"},{"issue":"5","key":"10564_CR36","doi-asserted-by":"publisher","first-page":"1771","DOI":"10.1109\/TSE.2020.3036108","volume":"48","author":"M Li","year":"2022","unstructured":"Li M, Chen T, Yao X (2022) How to evaluate solutions in pareto-based search-based software engineering: a critical review and methodological guidance. IEEE Trans Software Eng 48(5):1771\u20131799. https:\/\/doi.org\/10.1109\/TSE.2020.3036108","journal-title":"IEEE Trans Software Eng"},{"key":"10564_CR37","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1007\/978-3-030-88106-1_4","volume-title":"Search-based software engineering","author":"C Lu","year":"2021","unstructured":"Lu C, Zhang H, Yue T, Ali S (2021) Search-based selection and prioritization of test scenarios for autonomous driving systems. In: O\u2019Reilly UM, Devroey X (eds) Search-based software engineering. Springer International Publishing, Cham, pp 41\u201355"},{"key":"10564_CR38","doi-asserted-by":"publisher","unstructured":"Marculescu B, Feldt R, Torkar R (2016) Using exploration focused techniques to augment search-based software testing: an experimental evaluation. ICST, pp 69\u201379, https:\/\/doi.org\/10.1109\/ICST.2016.26","DOI":"10.1109\/ICST.2016.26"},{"key":"10564_CR39","doi-asserted-by":"crossref","unstructured":"Matinnejad R, Nejati S, Briand LC, Bruckmann T (2014) MiL testing of highly configurable continuous controllers: scalable search using surrogate models. In: ASE 2014, ACM, pp 163\u2013174","DOI":"10.1145\/2642937.2642978"},{"key":"10564_CR40","doi-asserted-by":"crossref","unstructured":"Matinnejad R, Nejati S, Briand LC, Bruckmann T, Poull C (2013) Automated model-in-the-loop testing of continuous controllers using search. In: Ruhe G, Zhang Y (eds) Search Based Software Engineering - 5th International Symposium, SSBSE 2013, St. Petersburg, Russia, August 24-26, 2013. Proceedings, Springer, Lecture Notes in Computer Science, vol 8084, pp 141\u2013157","DOI":"10.1007\/978-3-642-39742-4_12"},{"key":"10564_CR41","doi-asserted-by":"crossref","unstructured":"McKay MD, Beckman RJ, Conover WJ (1979) A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21(2):239\u2013245, http:\/\/www.jstor.org\/stable\/1268522","DOI":"10.1080\/00401706.1979.10489755"},{"key":"10564_CR42","doi-asserted-by":"publisher","unstructured":"Moghadam MH, Borg M, Mousavirad SJ (2021) Deeper at the sbst 2021 tool competition: Adas testing using multi-objective search. In: SBST, pp 40\u201341, https:\/\/doi.org\/10.1109\/SBST52555.2021.00018","DOI":"10.1109\/SBST52555.2021.00018"},{"key":"10564_CR43","doi-asserted-by":"crossref","unstructured":"Moghadam MH, Borg M, Saadatmand M, Mousavirad SJ, Bohlin M, Lisper B (2023) Machine learning testing in an adas case study using simulation-integrated bio-inspired sbt. J Softw: Evol Process p e2591","DOI":"10.1002\/smr.2591"},{"key":"10564_CR44","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1007\/978-3-642-18272-3_10","volume-title":"New horizons in evolutionary robotics","author":"JB Mouret","year":"2011","unstructured":"Mouret JB (2011) Novelty-based multiobjectivization. In: Doncieux S, Bred\u00e8che N, Mouret JB (eds) New horizons in evolutionary robotics. Springer, Berlin Heidelberg, Berlin, Heidelberg, pp 139\u2013154"},{"key":"10564_CR45","unstructured":"Mouret J, Clune J (2015) Illuminating search spaces by mapping elites. CoRR abs\/1504.04909,"},{"key":"10564_CR46","doi-asserted-by":"publisher","unstructured":"Nabhan M, Schoenauer M, Tourbier Y, Hage H (2019) Optimizing coverage of simulated driving scenarios for the autonomous vehicle. ICCVE, pp 1\u20135, https:\/\/doi.org\/10.1109\/ICCVE45908.2019.8965211","DOI":"10.1109\/ICCVE45908.2019.8965211"},{"key":"10564_CR47","doi-asserted-by":"publisher","unstructured":"Neelofar N, Aleti A (2024a) Identifying and explaining safety-critical scenarios for autonomous vehicles via key features. ACM Trans Softw Eng Methodol 33(4):1\u201332, https:\/\/doi.org\/10.1145\/3640335","DOI":"10.1145\/3640335"},{"key":"10564_CR48","doi-asserted-by":"publisher","unstructured":"Neelofar N, Aleti A (2024b) Towards reliable ai: adequacy metrics for ensuring the quality of system-level testing of autonomous vehicles. In: Proceedings of the IEEE\/ACM 46th international conference on software engineering, Association for Computing Machinery, New York, NY, USA, ICSE \u201924, pp 1\u201312, https:\/\/doi.org\/10.1145\/3597503.3623314","DOI":"10.1145\/3597503.3623314"},{"key":"10564_CR49","doi-asserted-by":"crossref","unstructured":"Nejati S, Sorokin L, Safin D, Formica F, Mahboob MM, Menghi C (2023) Reflections on surrogate-assisted search-based testing: a taxonomy and two replication studies based on industrial adas and simulink models. Inf Softw Technol 163:107286","DOI":"10.1016\/j.infsof.2023.107286"},{"key":"10564_CR50","doi-asserted-by":"publisher","unstructured":"Ramakrishna S, Luo B, Barve Y, Karsai G, Dubey A (2022) Risk-aware scene sampling for dynamic assurance of autonomous systems. ICAA pp 107\u2013116, https:\/\/doi.org\/10.1109\/ICAA52185.2022.00022","DOI":"10.1109\/ICAA52185.2022.00022"},{"key":"10564_CR51","doi-asserted-by":"publisher","unstructured":"Riccio V, Tonella P (2020) Model-based exploration of the frontier of behaviours for deep learning system testing. In: Proceedings of the 28th ACM Joint meeting on European software engineering conference and symposium on the foundations of software engineering, Association for Computing Machinery, New York, NY, USA, ESEC\/FSE 2020, p 876\u2013888, https:\/\/doi.org\/10.1145\/3368089.3409730","DOI":"10.1145\/3368089.3409730"},{"key":"10564_CR52","doi-asserted-by":"publisher","unstructured":"Riccio V, Tonella P (2023) When and why test generators for deep learning produce invalid inputs: an empirical study. In: Proceedings of the 45th International Conference on Software Engineering, IEEE Press, ICSE \u201923, p 1161\u20131173, https:\/\/doi.org\/10.1109\/ICSE48619.2023.00104","DOI":"10.1109\/ICSE48619.2023.00104"},{"key":"10564_CR53","doi-asserted-by":"publisher","first-page":"10031","DOI":"10.1109\/ACCESS.2022.3142859","volume":"10","author":"TM Shami","year":"2022","unstructured":"Shami TM, El-Saleh AA, Alswaitti M, Al-Tashi Q, Summakieh MA, Mirjalili S (2022) Particle swarm optimization: a comprehensive survey. IEEE Access 10:10031\u201310061. https:\/\/doi.org\/10.1109\/ACCESS.2022.3142859","journal-title":"IEEE Access"},{"key":"10564_CR54","doi-asserted-by":"publisher","unstructured":"Shi Y, Eberhart R (1999) Empirical study of particle swarm optimization. In: Proceedings of the 1999 congress on evolutionary computation-CEC99 (Cat. No. 99TH8406), vol\u00a03, pp 1945\u20131950 vol. 3, https:\/\/doi.org\/10.1109\/CEC.1999.785511","DOI":"10.1109\/CEC.1999.785511"},{"key":"10564_CR55","unstructured":"Siemens (2023) Simcenter prescan. https:\/\/plm.sw.siemens.com\/de-DE\/simcenter\/autonomous-vehicle-solutions\/prescan\/"},{"key":"10564_CR56","doi-asserted-by":"publisher","first-page":"505","DOI":"10.1007\/978-3-540-31880-4_35","volume-title":"Evolutionary multi-criterion optimization","author":"MR Sierra","year":"2005","unstructured":"Sierra MR, Coello Coello CA (2005) Improving pso-based multi-objective optimization using crowding, mutation and e-dominance. In: Coello Coello CA, Hern\u00e1ndez Aguirre A, Zitzler E (eds) Evolutionary multi-criterion optimization. Springer, Berlin Heidelberg, Berlin, Heidelberg, pp 505\u2013519"},{"key":"10564_CR57","doi-asserted-by":"publisher","unstructured":"Sorokin L, Kerscher N (2024) Guiding the search towards failure-inducing test inputs using support vector machines. In: Proceedings of the 5th IEEE\/ACM international workshop on deep learning for testing and testing for deep learning, Association for Computing Machinery, New York, NY, USA, DeepTest \u201924, p 9\u201312, https:\/\/doi.org\/10.1145\/3643786.3648023","DOI":"10.1145\/3643786.3648023"},{"key":"10564_CR58","doi-asserted-by":"publisher","unstructured":"Sorokin L, Munaro T, Safin D, Liao BHC, Molin A (2024) OpenSBT: a modular framework for search-based testing of automated driving systems. In: Proceedings of the 2024 IEEE\/ACM 46th international conference on software engineering: companion proceedings, Association for Computing Machinery, New York, NY, USA, ICSE-Companion \u201924, p 94\u201398, https:\/\/doi.org\/10.1145\/3639478.3640027","DOI":"10.1145\/3639478.3640027"},{"key":"10564_CR59","unstructured":"Surjanovic S, Bingham D (2023) Virtual library of simulation experiments: test functions and datasets. Retrieved August 4, 2023, from http:\/\/www.sfu.ca\/~ssurjano"},{"key":"10564_CR60","doi-asserted-by":"publisher","unstructured":"Ulbrich S, Menzel T, Reschka A, Schuldt F, Maurer M (2015) Defining and substantiating the terms scene, situation, and scenario for automated driving. In: 2015 IEEE 18th International conference on intelligent transportation systems, pp 982\u2013988, https:\/\/doi.org\/10.1109\/ITSC.2015.164","DOI":"10.1109\/ITSC.2015.164"},{"key":"10564_CR61","doi-asserted-by":"publisher","unstructured":"Zeller A (2017) Search-based testing and system testing: a marriage in heaven. In: 2017 IEEE\/ACM 10th International workshop on Search-Based Software Testing (SBST), SBST, pp 49\u201350, https:\/\/doi.org\/10.1109\/SBST.2017.3","DOI":"10.1109\/SBST.2017.3"},{"key":"10564_CR62","doi-asserted-by":"publisher","unstructured":"Zohdinasab T, Riccio V, Gambi A, Tonella P (2021) Deephyperion: exploring the feature space of deep learning-based systems through illumination search. In: Proceedings of the 30th ACM SIGSOFT international symposium on software testing and analysis, Association for Computing Machinery, New York, NY, USA, ISSTA 2021, p 79\u201390, https:\/\/doi.org\/10.1145\/3460319.3464811","DOI":"10.1145\/3460319.3464811"},{"key":"10564_CR63","doi-asserted-by":"crossref","unstructured":"Zohdinasab T, Riccio V, Tonella P (2023) Deepatash: focused test generation for deep learning systems. In: Proceedings of the 32nd ACM SIGSOFT international symposium on software testing and analysis, Association for Computing Machinery, New York, NY, USA, ISSTA 2023, p 954\u2013966","DOI":"10.1145\/3597926.3598109"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-024-10564-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-024-10564-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-024-10564-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,1,7]],"date-time":"2025-01-07T14:13:17Z","timestamp":1736259197000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-024-10564-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,16]]},"references-count":63,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,1]]}},"alternative-id":["10564"],"URL":"https:\/\/doi.org\/10.1007\/s10664-024-10564-3","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,11,16]]},"assertion":[{"value":"1 October 2024","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 November 2024","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no competing interests to declare that are relevant to the content of this article.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing of Interests"}}],"article-number":"26"}}