{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T16:40:41Z","timestamp":1783096841255,"version":"3.54.6"},"reference-count":102,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2026,2,20]],"date-time":"2026-02-20T00:00:00Z","timestamp":1771545600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,2,20]],"date-time":"2026-02-20T00:00:00Z","timestamp":1771545600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100005713","name":"Technische Universit\u00e4t M\u00fcnchen","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100005713","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2026,7]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Scenario-based testing with driving simulators is extensively used to identify failing conditions of automated driving assistance systems (ADAS) and reduce the amount of in-field road testing. However, existing studies have shown that repeated test execution in the same as well as in distinct simulators can yield different outcomes, which can be attributed to sources of flakiness or different implementations of the physics, among other factors. In this paper, we present , a novel approach to multi-simulation ADAS testing based on a search-based testing approach that leverages an ensemble of simulators to identify failure-inducing, simulator-agnostic test scenarios. During the search, each scenario is evaluated jointly on multiple simulators. Scenarios that produce consistent results across simulators are prioritized for further exploration, while those that fail on only a subset of simulators are given less priority, as they may reflect simulator-specific issues rather than generalizable failures. Our empirical study, which involves testing three lane-keeping ADAS with increasing complexity on different pairs of three widely used simulators, demonstrates that  outperforms single-simulator testing by achieving, on average, a higher rate of simulator-agnostic failures by 66%. Compared to a state-of-the-art multi-simulator approach that combines the outcome of independent test generation campaigns obtained in different simulators,  identifies on average up to\n                    <jats:inline-formula>\n                      <jats:tex-math>$$3.4\\times$$<\/jats:tex-math>\n                    <\/jats:inline-formula>\n                    more simulator-agnostic failing tests and higher failure rates. To avoid the costly execution of test inputs on which simulators disagree, we propose an enhancement of  that leverages surrogate models to predict simulator disagreements and bypass test executions. Our results show that utilizing a surrogate model during the search does not only retain the average number of valid failures but also improves its efficiency in finding the first valid failure. These findings indicate that combining an ensemble of simulators during the search is a promising approach for the automated cross-replication in ADAS testing.\n                  <\/jats:p>","DOI":"10.1007\/s10664-026-10821-7","type":"journal-article","created":{"date-parts":[[2026,2,20]],"date-time":"2026-02-20T11:11:45Z","timestamp":1771585905000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Simulator ensembles for trustworthy autonomous driving systems testing"],"prefix":"10.1007","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-1162-6252","authenticated-orcid":false,"given":"Lev","family":"Sorokin","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7825-3409","authenticated-orcid":false,"given":"Matteo","family":"Biagiola","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8956-3894","authenticated-orcid":false,"given":"Andrea","family":"Stocco","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,2,20]]},"reference":[{"key":"10821_CR1","doi-asserted-by":"publisher","unstructured":"Abdessalem RB, Nejati S, Briand LC, Stifter T (2018) Testing vision-based control systems using learnable evolutionary algorithms. In: Chaudron M, Crnkovic I, Chechik M, Harman M (eds), Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018, pp 1016\u20131026. ACM. https:\/\/doi.org\/10.1145\/3180155.3180160","DOI":"10.1145\/3180155.3180160"},{"key":"10821_CR2","doi-asserted-by":"publisher","unstructured":"Abdessalem RB, Panichella A, Nejati S, Briand LC, Stifter T (2018) Testing autonomous cars for feature interaction failures using many-objective search. In: Proceedings of ASE \u201918, ASE 2018, pp 143\u2013154. ACM. https:\/\/doi.org\/10.1145\/3238147.3238192","DOI":"10.1145\/3238147.3238192"},{"key":"10821_CR3","doi-asserted-by":"publisher","unstructured":"Afzal A, Katz DS, Le Goues C, Timperley CS (2021) Simulation for robotics test automation: Developer perspectives. In: 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST), pp 263\u2013274. IEEE. https:\/\/doi.org\/10.1109\/ICST49551.2021.00036","DOI":"10.1109\/ICST49551.2021.00036"},{"key":"10821_CR4","doi-asserted-by":"crossref","unstructured":"Ali Q, Stocco A, Mariani L, Riganelli O (2025) OpenCat: Improving Interoperability of ADS Testing. In: Proceedings of 47th International Conference on Software Engineering Workshops, ICSEW \u201924, p 10 pages. IEEE","DOI":"10.1109\/DeepTest66595.2025.00013"},{"key":"10821_CR5","doi-asserted-by":"publisher","unstructured":"Amini MH, Naseri S, Nejati S (2024) Evaluating the impact of flaky simulators on testing autonomous driving systems. Empirical Softw Engg 29(2). https:\/\/doi.org\/10.1007\/s10664-023-10433-5","DOI":"10.1007\/s10664-023-10433-5"},{"key":"10821_CR6","doi-asserted-by":"crossref","unstructured":"Ando A, Gidaris S, Bursuc A, Puy G, Boulch A, Marlet R (2023) Rangevit: Towards vision transformers for 3d semantic segmentation in autonomous driving. In: CVPR","DOI":"10.1109\/CVPR52729.2023.00507"},{"key":"10821_CR7","doi-asserted-by":"publisher","unstructured":"Arrieta A, Wang S, Markiegi U, Sagardui G, Etxeberria L (2017) Search-based test case generation for cyber-physical systems. In: 2017 IEEE Congress on Evolutionary Computation (CEC), pp 688\u2013697. https:\/\/doi.org\/10.1109\/CEC.2017.7969377","DOI":"10.1109\/CEC.2017.7969377"},{"key":"10821_CR8","unstructured":"Autoware (2024). https:\/\/autoware.org"},{"key":"10821_CR9","unstructured":"Baidu (2024) Baidu Apollo. https:\/\/github.com\/ApolloAuto\/apollo\/"},{"key":"10821_CR10","unstructured":"Baidu Inc (2018) Baidu Apolloscapes Dataset. https:\/\/apolloscape.auto\/index.html. Accessed: [2024-01-15]"},{"key":"10821_CR11","doi-asserted-by":"crossref","unstructured":"Baresi L, Hu DYX, Stocco A, Tonella P (2025) Efficient domain augmentation for autonomous driving testing using diffusion models. In: Proceedings of 47th International Conference on Software Engineering, ICSE \u201925. IEEE","DOI":"10.1109\/ICSE55347.2025.00206"},{"key":"10821_CR12","unstructured":"BeamNG.research (2022) BeamNG GmbH. https:\/\/www.beamng.gmbh\/research. Online; accessed 18\/08\/2019"},{"key":"10821_CR13","doi-asserted-by":"publisher","unstructured":"Ben Abdessalem R, Nejati S, Briand LC, Stifter T (2016) Testing advanced driver assistance systems using multi-objective search and neural networks. ASE, pp 63\u201374. https:\/\/doi.org\/10.1145\/2970276.2970311","DOI":"10.1145\/2970276.2970311"},{"key":"10821_CR14","doi-asserted-by":"crossref","unstructured":"Ben Abdessalem R, Nejati S, Briand LC, Stifter T (2016) Testing advanced driver assistance systems using multi-objective search and neural networks. In: 2016 31st IEEE\/ACM International Conference on Automated Software Engineering (ASE), pp 63\u201374","DOI":"10.1145\/2970276.2970311"},{"key":"10821_CR15","doi-asserted-by":"publisher","unstructured":"Biagiola M, Klikovits S (2024) Sbft tool competition 2024 - cyber-physical systems track. In: Proceedings of the 17th ACM\/IEEE International Workshop on Search-Based and Fuzz Testing, SBFT \u201924, pp 33\u201336. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3643659.3643932","DOI":"10.1145\/3643659.3643932"},{"key":"10821_CR16","doi-asserted-by":"publisher","unstructured":"Biagiola M, Stocco A, Riccio V, Tonella P (2024) Two is better than one: digital siblings to improve autonomous driving testing. Empirical Softw Engg 29(4). https:\/\/doi.org\/10.1007\/s10664-024-10458-4","DOI":"10.1007\/s10664-024-10458-4"},{"key":"10821_CR17","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3420816","author":"M Biagiola","year":"2023","unstructured":"Biagiola M, Tonella P (2023). Boundary state generation for testing and improvement of autonomous driving systems. https:\/\/doi.org\/10.1109\/TSE.2024.3420816","journal-title":"Boundary state generation for testing and improvement of autonomous driving systems"},{"issue":"3","key":"10821_CR18","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1007\/S10664-023-10286-Y","volume":"28","author":"C Birchler","year":"2023","unstructured":"Birchler C, Khatiri S, Bosshard B, Gambi A, Panichella S (2023) Machine learning-based test selection for simulation-based testing of self-driving cars software. Empir Softw Eng 28(3):71. https:\/\/doi.org\/10.1007\/S10664-023-10286-Y","journal-title":"Empir Softw Eng"},{"key":"10821_CR19","unstructured":"Bojarski M, Del Testa D, Dworakowski D, Firner B, Flepp B, Goyal P, Jackel LD, Monfort M, Muller U, Zhang J, Zhang X, Zhao J, Zieba K (2016) End to end learning for self-driving cars. arXiv:1604.07316"},{"key":"10821_CR20","doi-asserted-by":"publisher","unstructured":"Borg M, Abdessalem R, Nejati S, Jegeden F, Shin D (2021) Digital twins are not monozygotic \u2013 cross-replicating adas testing in two industry-grade automotive simulators pp 383\u2013393. https:\/\/doi.org\/10.1109\/ICST49551.2021.00050","DOI":"10.1109\/ICST49551.2021.00050"},{"issue":"1","key":"10821_CR21","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1023\/A:1010933404324","volume":"45","author":"L Breiman","year":"2001","unstructured":"Breiman L (2001) Random forests. Mach Learn 45(1):5\u201332. https:\/\/doi.org\/10.1023\/A:1010933404324","journal-title":"Mach Learn"},{"key":"10821_CR22","unstructured":"Breiman L, Friedman J, Stone C, Olshen R (1984) Classification and Regression Trees. Taylor & Francis. https:\/\/books.google.de\/books?id=JwQx-WOmSyQC"},{"key":"10821_CR23","unstructured":"Cabrera A (1994) Logistic regression analysis in higher education: An applied perspective. Higher education: Handbook of theory and research X\/Agathon Press"},{"issue":"2","key":"10821_CR24","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1145\/3177753","volume":"61","author":"VG Cerf","year":"2018","unstructured":"Cerf VG (2018) A comprehensive self-driving car test. Commun ACM 61(2):7\u20137. https:\/\/doi.org\/10.1145\/3177753","journal-title":"Commun ACM"},{"key":"10821_CR25","doi-asserted-by":"crossref","unstructured":"Chen X, Biagiola M, Riccio V, d\u2019Amorim M, Stocco A (2025) XMutant: XAI-based Fuzzing for Deep Learning Systems (2025). arXiv:2503.07222","DOI":"10.1007\/s10664-025-10792-1"},{"key":"10821_CR26","doi-asserted-by":"publisher","unstructured":"Cheng M, Zhou Y, Xie X (2023) Behavexplor: Behavior diversity guided testing for autonomous driving systems. In: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2023, pp 488\u2013500. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3597926.3598072","DOI":"10.1145\/3597926.3598072"},{"key":"10821_CR27","doi-asserted-by":"crossref","unstructured":"Codevilla F, L\u00f3pez AM, Koltun V, Dosovitskiy A (2018) On offline evaluation of vision-based driving models. arXiv:1809.04843","DOI":"10.1007\/978-3-030-01267-0_15"},{"issue":"3","key":"10821_CR28","doi-asserted-by":"publisher","first-page":"273","DOI":"10.1023\/A:1022627411411","volume":"20","author":"C Cortes","year":"1995","unstructured":"Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273\u2013297. https:\/\/doi.org\/10.1023\/A:1022627411411","journal-title":"Mach Learn"},{"key":"10821_CR29","unstructured":"Donkey Car (2021). https:\/\/www.donkeycar.com\/"},{"key":"10821_CR30","unstructured":"Dosovitskiy A, Ros G, Codevilla F, L\u00f3pez A, Koltun V (2017) CARLA: an open urban driving simulator. arXiv:1711.03938"},{"key":"10821_CR31","doi-asserted-by":"crossref","unstructured":"Friedman JH (2002) Stochastic gradient boosting. Computational Statistics & Data Analysis 38(4):367\u2013378. https:\/\/doi.org\/10.1016\/S0167-9473(01)00065-2.https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0167947301000652. Nonlinear Methods and Data Mining","DOI":"10.1016\/S0167-9473(01)00065-2"},{"key":"10821_CR32","doi-asserted-by":"publisher","unstructured":"Gambi A, Jahangirova G, Riccio V, Zampetti F (2023) Sbst tool competition 2022. In: Proceedings of the 15th Workshop on Search-Based Software Testing, SBST \u201922, pp 25\u201332. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3526072.3527538","DOI":"10.1145\/3526072.3527538"},{"key":"10821_CR33","doi-asserted-by":"crossref","unstructured":"Gambi A, Mueller M, Fraser G (2019) Automatically testing self-driving cars with search-based procedural content generation. In: Proceedings of ISSTA \u201919","DOI":"10.1145\/3293882.3330566"},{"key":"10821_CR34","doi-asserted-by":"publisher","unstructured":"Garc\u00eda S, Str\u00fcber D, Brugali D, Berger T, Pelliccione P (2020) Robotics software engineering: A perspective from the service robotics domain. In: Proceedings of ESEC\/FSE \u201920, ESEC\/FSE 2020, pp 593\u2013604. ACM, USA. https:\/\/doi.org\/10.1145\/3368089.3409743","DOI":"10.1145\/3368089.3409743"},{"key":"10821_CR35","doi-asserted-by":"crossref","unstructured":"Grewal R, Tonella P, Stocco A (2024) Predicting Safety Misbehaviours in Autonomous Driving Systems using Uncertainty Quantification. In: Proceedings of 17th IEEE International Conference on Software Testing, Verification and Validation, ICST \u201924, p 12 pages. IEEE","DOI":"10.1109\/ICST60714.2024.00016"},{"key":"10821_CR36","unstructured":"Group E (2021) Esi prosivic. https:\/\/myesi.esi-group.com\/downloads\/software-downloads\/pro-sivic-2021.0"},{"key":"10821_CR37","doi-asserted-by":"publisher","unstructured":"Haq FU, Shin D, Nejati S, Briand L (2021) Can offline testing of deep neural networks replace their online testing? a case study of automated driving systems. Empirical Softw Engg 26(5). https:\/\/doi.org\/10.1007\/s10664-021-09982-4","DOI":"10.1007\/s10664-021-09982-4"},{"key":"10821_CR38","doi-asserted-by":"publisher","unstructured":"Humbatova N, Jahangirova G, Bavota G, Riccio V, Stocco A, Tonella P (2020) Taxonomy of Real Faults in Deep Learning Systems. In: Proceedings of 42nd International Conference on Software Engineering, ICSE\u201920, p 12 pages. ACM, New York, NY, USA. https:\/\/doi.org\/10.1145\/3377811.3380395","DOI":"10.1145\/3377811.3380395"},{"key":"10821_CR39","doi-asserted-by":"crossref","unstructured":"Humeniuk D, Khomh F, Antoniol G (2022) A search-based framework for automatic generation of testing environments for cyber\u2013physical systems. Inf Softw Technol 149, 106,936. https:\/\/doi.org\/10.1016\/j.infsof.2022.106936, https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0950584922000866","DOI":"10.1016\/j.infsof.2022.106936"},{"key":"10821_CR40","doi-asserted-by":"publisher","unstructured":"Humeniuk D, Khomh F, Antoniol G (2024) Reinforcement learning informed evolutionary search for autonomous systems testing. ACM Trans Softw Eng Methodol. https:\/\/doi.org\/10.1145\/3680468. Just Accepted","DOI":"10.1145\/3680468"},{"key":"10821_CR41","doi-asserted-by":"publisher","unstructured":"Hussain M, Ali N, Hong JE (2022) Deepguard: A framework for safeguarding autonomous driving systems from inconsistent behaviour. Automated Software Engg 29(1). https:\/\/doi.org\/10.1007\/s10515-021-00310-0","DOI":"10.1007\/s10515-021-00310-0"},{"key":"10821_CR42","doi-asserted-by":"crossref","unstructured":"Jahangirova G, Stocco A, Tonella P (2021) Quality metrics and oracles for autonomous vehicles testing. In: Proceedings of 14th IEEE international conference on software testing, verification and validation, ICST \u201921. IEEE","DOI":"10.1109\/ICST49551.2021.00030"},{"key":"10821_CR43","doi-asserted-by":"crossref","unstructured":"Jha S, Banerjee SS, Tsai T, Hari SKS, Sullivan MB, Kalbarczyk ZT, Keckler SW, Iyer RK (2019) Ml-based fault injection for autonomous vehicles: A case for bayesian fault injection. 2019 49th Annual IEEE\/IFIP International Conference on Dependable Systems and Networks (DSN) pp 112\u2013124. https:\/\/api.semanticscholar.org\/CorpusID:195776612","DOI":"10.1109\/DSN.2019.00025"},{"key":"10821_CR44","doi-asserted-by":"publisher","unstructured":"Jodat BA, Chandar A, Nejati S, Sabetzadeh M (2024) Test generation strategies for building failure models and explaining spurious failures. ACM Trans Softw Eng Methodol 33(4). https:\/\/doi.org\/10.1145\/3638246","DOI":"10.1145\/3638246"},{"key":"10821_CR45","doi-asserted-by":"publisher","unstructured":"Kim J, Feldt R, Yoo S (2019) Guiding deep learning system testing using surprise adequacy. In: Proceedings of the 41st international conference on software engineering, ICSE \u201919, pp 1039\u20131049. IEEE Press, Piscataway, NJ, USA. https:\/\/doi.org\/10.1109\/ICSE.2019.00108","DOI":"10.1109\/ICSE.2019.00108"},{"key":"10821_CR46","doi-asserted-by":"publisher","unstructured":"Kim S, Liu M, Rhee JJ, Jeon Y, Kwon Y, Kim CH (2022) DriveFuzz. In: Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security. ACM. https:\/\/doi.org\/10.1145\/3548606.3560558","DOI":"10.1145\/3548606.3560558"},{"key":"10821_CR47","doi-asserted-by":"publisher","unstructured":"Kl\u00fcck F, Zimmermann M, Wotawa F, Nica M (2019) Genetic algorithm-based test parameter optimization for adas system testing. In: QRS, pp 418\u2013425. https:\/\/doi.org\/10.1109\/QRS.2019.00058","DOI":"10.1109\/QRS.2019.00058"},{"key":"10821_CR48","unstructured":"Kolesnikov A, Dosovitskiy A, Weissenborn D, Heigold G, Uszkoreit J, Beyer L, Minderer M, Dehghani M, Houlsby N, Gelly S, Unterthiner T, Zhai X (2021) An image is worth 16x16 words: Transformers for image recognition at scale"},{"key":"10821_CR49","doi-asserted-by":"publisher","unstructured":"Koroglu Y, Wotawa F (2023) Towards a review on simulated adas\/ad testing. In: 2023 IEEE\/ACM International Conference on Automation of Software Test (AST), pp 112\u2013122. https:\/\/doi.org\/10.1109\/AST58925.2023.00015","DOI":"10.1109\/AST58925.2023.00015"},{"key":"10821_CR50","doi-asserted-by":"crossref","unstructured":"Lambertenghi SC, Leonhard H, Stocco A (2025a) Benchmarking image perturbations for testing automated driving assistance systems. In: Proceedings of the IEEE 18th IEEE International Conference on Software Testing, Verification and Validation, ICST \u201925, p 12 pages. IEEE","DOI":"10.1109\/ICST62969.2025.10988980"},{"key":"10821_CR51","doi-asserted-by":"crossref","unstructured":"Lambertenghi SC, Stocco A (2024) Assessing quality metrics for neural reality gap input mitigation in autonomous driving testing. In: Proceedings of 17th IEEE International Conference on Software Testing, Verification and Validation, ICST \u201924, p 12 pages. IEEE","DOI":"10.1109\/ICST60714.2024.00024"},{"key":"10821_CR52","doi-asserted-by":"crossref","unstructured":"Lambertenghi SC, Valdez MF, Stocco A (2025b) A multi-modality evaluation of the reality gap in autonomous driving systems. In: Proceedings of the 40th IEEE\/ACM international conference on automated software engineering, ASE \u201925, p 12 pages. IEEE","DOI":"10.1109\/ASE63991.2025.00230"},{"key":"10821_CR53","doi-asserted-by":"crossref","unstructured":"Li D, Auerbach P, Okhrin O (2024) Autonomous driving small-scale cars: A survey of recent development. arXiv:2404.06229","DOI":"10.1109\/TITS.2025.3572394"},{"key":"10821_CR54","doi-asserted-by":"publisher","unstructured":"Li G, Li Y, Jha S, Tsai T, Sullivan M, Hari SKS, Kalbarczyk Z, Iyer R (2020) Av-fuzzer: Finding safety violations in autonomous driving systems. In: 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE), pp 25\u201336. https:\/\/doi.org\/10.1109\/ISSRE5003.2020.00012","DOI":"10.1109\/ISSRE5003.2020.00012"},{"issue":"5","key":"10821_CR55","doi-asserted-by":"publisher","first-page":"1771","DOI":"10.1109\/TSE.2020.3036108","volume":"48","author":"M Li","year":"2022","unstructured":"Li M, Chen T, Yao X (2022) How to evaluate solutions in pareto-based search-based software engineering: A critical review and methodological guidance. IEEE Trans Software Eng 48(5):1771\u20131799. https:\/\/doi.org\/10.1109\/TSE.2020.3036108","journal-title":"IEEE Trans Software Eng"},{"issue":"2","key":"10821_CR56","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3300148","volume":"52","author":"M Li","year":"2019","unstructured":"Li M, Yao X (2019) Quality evaluation of solution sets in multiobjective optimisation: A survey. ACM Comput Surv 52(2):1\u201338. https:\/\/doi.org\/10.1145\/3300148","journal-title":"ACM Comput Surv"},{"issue":"5","key":"10821_CR57","doi-asserted-by":"publisher","first-page":"4861","DOI":"10.1109\/tiv.2024.3374044","volume":"9","author":"Y Li","year":"2024","unstructured":"Li Y, Yuan W, Zhang S, Yan W, Shen Q, Wang C, Yang M (2024) Choose your simulator wisely: A review on open-source simulators for autonomous driving. IEEE Transactions on Intelligent Vehicles 9(5):4861\u20134876. https:\/\/doi.org\/10.1109\/tiv.2024.3374044","journal-title":"IEEE Transactions on Intelligent Vehicles"},{"key":"10821_CR58","doi-asserted-by":"publisher","unstructured":"Liu D, Metzman J, Bohme M, Chang O, Arya A (2023) SBFT Tool Competition 2023 - Fuzzing Track. In: 2023 IEEE\/ACM International Workshop on Search-Based and Fuzz Testing (SBFT), pp 51\u201354. IEEE Computer Society, Los Alamitos, CA, USA. https:\/\/doi.org\/10.1109\/SBFT59156.2023.00016","DOI":"10.1109\/SBFT59156.2023.00016"},{"key":"10821_CR59","doi-asserted-by":"publisher","unstructured":"Lou G, Deng Y, Zheng X, Zhang M, Zhang T (2022) Testing of autonomous driving systems: where are we and where should we go? In: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC\/FSE 2022, pp 31\u201343. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3540250.3549111","DOI":"10.1145\/3540250.3549111"},{"key":"10821_CR60","doi-asserted-by":"publisher","unstructured":"Lu C, Ali S, Yue T (2024) Epitester: Testing autonomous vehicles with epigenetic algorithm and attention mechanism. IEEE Trans Softw Eng pp 1\u201319. https:\/\/doi.org\/10.1109\/TSE.2024.3449429","DOI":"10.1109\/TSE.2024.3449429"},{"issue":"1","key":"10821_CR61","doi-asserted-by":"publisher","first-page":"384","DOI":"10.1109\/TSE.2022.3150788","volume":"49","author":"C Lu","year":"2023","unstructured":"Lu C, Shi Y, Zhang H, Zhang M, Wang T, Yue T, Ali S (2023) Learning configurations of operating environment of autonomous vehicles to maximize their collisions. IEEE Trans Software Eng 49(1):384\u2013402. https:\/\/doi.org\/10.1109\/TSE.2022.3150788","journal-title":"IEEE Trans Software Eng"},{"key":"10821_CR62","unstructured":"Lu C, Yue T, Zhang M, Ali S (2023) Deepqtest: Testing autonomous driving systems with reinforcement learning and real-world weather data. arXiv:2310.05170"},{"key":"10821_CR63","doi-asserted-by":"publisher","unstructured":"Marko N, Ruebsam J, Biehn A, Schneider H (2019) Scenario-based testing of adas - integration of the open simulation interface into co-simulation for function validation. In: Proceedings of the 9th International Conference on Simulation and Modeling Methodologies, Technologies and Applications - SIMULTECH, pp 255\u2013262. INSTICC, SciTePress. https:\/\/doi.org\/10.5220\/0007838302550262","DOI":"10.5220\/0007838302550262"},{"key":"10821_CR64","doi-asserted-by":"crossref","unstructured":"Matinnejad R, Nejati S, Briand L, Bruckmann T, Poull C (2015) Search-based automated testing of continuous controllers: Framework, tool support, and case studies. Inf Softw Technol 57:705\u2013722. https:\/\/doi.org\/10.1016\/j.infsof.2014.05.007, https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0950584914001244","DOI":"10.1016\/j.infsof.2014.05.007"},{"key":"10821_CR65","doi-asserted-by":"publisher","unstructured":"McMinn P (2011) Search-based software testing: Past, present and future. In: 2011 IEEE fourth international conference on software testing, verification and validation workshops, pp 153\u2013163. https:\/\/doi.org\/10.1109\/ICSTW.2011.100","DOI":"10.1109\/ICSTW.2011.100"},{"key":"10821_CR66","doi-asserted-by":"crossref","unstructured":"Moghadam MH, Borg M, Saadatmand M, Mousavirad SJ, Bohlin M, Lisper B (2023) Machine learning testing in an adas case study using simulation-integrated bio-inspired sbt. J. Softw.: Evol. Process p. e2591","DOI":"10.1002\/smr.2591"},{"key":"10821_CR67","doi-asserted-by":"publisher","unstructured":"Mokhtarian A, Xu J, Scheffe P, Kloock M, Sch\u00e4fer S, Bang H, Le VA, Ulhas S, Betz J, Wilson S, Berman S, Prorok A, Alrifaee B (2024) A survey on small-scale testbeds for connected and automated vehicles and robot swarms. https:\/\/doi.org\/10.13140\/RG.2.2.16176.74248\/1","DOI":"10.13140\/RG.2.2.16176.74248\/1"},{"key":"10821_CR68","unstructured":"Mouret JB, Clune J (2015) Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04909"},{"key":"10821_CR69","doi-asserted-by":"crossref","unstructured":"Mullins GE, Stankiewicz PG, Hawthorne RC, Gupta SK (2018) Adaptive generation of challenging scenarios for testing and evaluation of autonomous vehicles. J Syst Softw 137:197\u2013215. https:\/\/doi.org\/10.1016\/j.jss.2017.10.031. http:\/\/www.sciencedirect.com\/science\/article\/pii\/S0164121217302546","DOI":"10.1016\/j.jss.2017.10.031"},{"key":"10821_CR70","doi-asserted-by":"crossref","unstructured":"Nejati S, Sorokin L, Safin D, Formica F, Mahboob MM, Menghi C (2023) Reflections on surrogate-assisted search-based testing: A taxonomy and two replication studies based on industrial adas and simulink models. Inf Softw Technol 163, 107,286","DOI":"10.1016\/j.infsof.2023.107286"},{"key":"10821_CR71","unstructured":"Opletal J (2025) China\u2019s massive adas test: 36 cars, 15 hazard scenarios, 216 crashes. https:\/\/carnewschina.com\/2025\/07\/24\/chinas-massive-adas-test-36-cars-15-hazard-scenarios-216-crashes\/"},{"key":"10821_CR72","doi-asserted-by":"publisher","DOI":"10.1145\/3743673","author":"Q Pan","year":"2025","unstructured":"Pan Q, Wang T, Ma J, Arcaini P, Yue T (2025) Simulation-based safety assessment of vehicle characteristics variations in autonomous driving systems. ACM Trans Softw Eng Methodol. https:\/\/doi.org\/10.1145\/3743673","journal-title":"ACM Trans Softw Eng Methodol"},{"key":"10821_CR73","doi-asserted-by":"publisher","unstructured":"Parry O, Kapfhammer GM, Hilton M, McMinn P (2022) A survey of flaky tests. ACM Trans Softw Eng Methodol 31(1):17:1\u201317:74. https:\/\/doi.org\/10.1145\/3476105","DOI":"10.1145\/3476105"},{"key":"10821_CR74","doi-asserted-by":"publisher","unstructured":"Pei K, Cao Y, Yang J, Jana S (2017) Deepxplore: Automated whitebox testing of deep learning systems. In: Proceedings of the 26th symposium on operating systems principles, SOSP \u201917, pp 1\u201318. ACM, New York, NY, USA. https:\/\/doi.org\/10.1145\/3132747.3132785","DOI":"10.1145\/3132747.3132785"},{"key":"10821_CR75","doi-asserted-by":"crossref","unstructured":"Riccio V, Jahangirova G, Stocco A, Humbatova N, Weiss M, Tonella P (2020) Testing Machine Learning based Systems: A Systematic Mapping. Empir Softw Eng","DOI":"10.1007\/s10664-020-09881-0"},{"key":"10821_CR76","doi-asserted-by":"crossref","unstructured":"Riccio V, Tonella P (2020) Model-Based Exploration of the Frontier of Behaviours for Deep Learning System Testing. In: Proceedings of ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC\/FSE \u201920","DOI":"10.1145\/3368089.3409730"},{"key":"10821_CR77","doi-asserted-by":"publisher","unstructured":"Riccio V, Tonella P (2023) When and why test generators for deep learning produce invalid inputs: an empirical study. In: 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE), ICSE \u201923, pp 1161\u20131173. IEEE Press. https:\/\/doi.org\/10.1109\/ICSE48619.2023.00104","DOI":"10.1109\/ICSE48619.2023.00104"},{"key":"10821_CR78","unstructured":"Software SDI (2022) Simcenter prescan. https:\/\/www.plm.automation.siemens.com\/global\/en\/products\/simcenter\/prescan.html. https:\/\/plm.sw.siemens.com\/de-DE\/simcenter\/autonomous-vehicle-solutions\/prescan\/"},{"key":"10821_CR79","doi-asserted-by":"publisher","unstructured":"Sorokin L, Kerscher N (2024) Guiding the search towards failure-inducing test inputs using support vector machines. In: Proceedings of the 5th IEEE\/ACM international workshop on deep learning for testing and testing for deep learning, DeepTest \u201924, pp 9\u201312. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3643786.3648023","DOI":"10.1145\/3643786.3648023"},{"key":"10821_CR80","doi-asserted-by":"publisher","unstructured":"Sorokin L, Munaro T, Safin D, Liao BHC, Molin A (2023) OpenSBT: A modular framework for search-based testing of automated driving systems. https:\/\/doi.org\/10.1145\/3639478.3640027","DOI":"10.1145\/3639478.3640027"},{"issue":"1","key":"10821_CR81","doi-asserted-by":"publisher","first-page":"26","DOI":"10.1007\/s10664-024-10564-3","volume":"30","author":"L Sorokin","year":"2024","unstructured":"Sorokin L, Safin D, Nejati S (2024) Can search-based testing with pareto optimization effectively cover failure-revealing test inputs? Empir Softw Eng 30(1):26. https:\/\/doi.org\/10.1007\/s10664-024-10564-3","journal-title":"Empir Softw Eng"},{"key":"10821_CR82","doi-asserted-by":"publisher","unstructured":"Stocco A, Nunes PJ, d\u2019Amorim M, Tonella P (2022) ThirdEye: Attention Maps for Safe Autonomous Driving Systems. In: Proceedings of 37th IEEE\/ACM International Conference on Automated Software Engineering, ASE \u201922. IEEE\/ACM. https:\/\/doi.org\/10.1145\/3551349.3556968","DOI":"10.1145\/3551349.3556968"},{"issue":"04","key":"10821_CR83","doi-asserted-by":"publisher","first-page":"1928","DOI":"10.1109\/TSE.2022.3202311","volume":"49","author":"A Stocco","year":"2023","unstructured":"Stocco A, Pulfer B, Tonella P (2023) Mind the Gap! A Study on the Transferability of Virtual Versus Physical-World Testing of Autonomous Driving Systems. IEEE Trans Software Eng 49(04):1928\u20131940. https:\/\/doi.org\/10.1109\/TSE.2022.3202311","journal-title":"IEEE Trans Software Eng"},{"issue":"3","key":"10821_CR84","doi-asserted-by":"publisher","first-page":"73","DOI":"10.1007\/s10664-023-10306-x","volume":"28","author":"A Stocco","year":"2023","unstructured":"Stocco A, Pulfer B, Tonella P (2023) Model vs system level testing of autonomous driving systems: A replication and extension study. Empirical Softw Engg 28(3):73. https:\/\/doi.org\/10.1007\/s10664-023-10306-x","journal-title":"Empirical Softw Engg"},{"key":"10821_CR85","doi-asserted-by":"publisher","DOI":"10.1002\/smr.2386","author":"A Stocco","year":"2021","unstructured":"Stocco A, Tonella P (2021) Confidence-driven weighted retraining for predicting safety-critical failures in autonomous driving systems. Journal of Software Evolution and Process. https:\/\/doi.org\/10.1002\/smr.2386","journal-title":"Journal of Software Evolution and Process"},{"key":"10821_CR86","doi-asserted-by":"crossref","unstructured":"Stocco A, Weiss M, Calzana M, Tonella P (2020) Misbehaviour prediction for autonomous driving systems. In: Proceedings of 42nd International Conference on Software Engineering, ICSE \u201920, p 12 pages. ACM","DOI":"10.1145\/3377811.3380353"},{"key":"10821_CR87","doi-asserted-by":"publisher","unstructured":"Tang S, Zhang Z, Zhang Y, Zhou J, Guo Y, Liu S, Guo S, Li Y, Ma L, Xue Y, Liu Y (2023) A survey on automated driving system testing: Landscapes and trends. ACM Trans Softw Eng Methodol 32(5):124:1\u2013124:62. https:\/\/doi.org\/10.1145\/3579642","DOI":"10.1145\/3579642"},{"key":"10821_CR88","unstructured":"Tawn Kramer ME (2022) contributors: Donkeycar. https:\/\/www.donkeycar.com\/"},{"key":"10821_CR89","doi-asserted-by":"publisher","unstructured":"Tian Y, Pei K, Jana S, Ray B (2018) Deeptest: Automated testing of deep-neural-network-driven autonomous cars. In: Proceedings of ICSE \u201918, ICSE \u201918, pp 303\u2013314. ACM, New York, NY, USA. https:\/\/doi.org\/10.1145\/3180155.3180220","DOI":"10.1145\/3180155.3180220"},{"key":"10821_CR90","unstructured":"Tier IV Inc (2023) Awsim: Autonomous driving simulator by tier iv. https:\/\/github.com\/tier4\/AWSIM. Accessed: 2025\u201311-03"},{"key":"10821_CR91","unstructured":"Replication package (2025). https:\/\/github.com\/ast-fortiss-tum\/MultiSim"},{"key":"10821_CR92","unstructured":"of Transportation UD (2018) A framework for automated driving system testable cases and scenarios. https:\/\/rosap.ntl.bts.gov\/view\/dot\/38824\/dot_38824_DS1.pdf"},{"key":"10821_CR93","unstructured":"of Transportation UD (2022) Standing general order on crash reporting for level 2 advanced driver assistance systems. https:\/\/www.nhtsa.gov\/sites\/nhtsa.gov\/files\/2022-06\/ADAS-L2-SGO-Report-June-2022.pdf"},{"key":"10821_CR94","unstructured":"Udacity (2017) A self-driving car simulator built with Unity. https:\/\/github.com\/udacity\/self-driving-car-sim. Online; accessed 18 August 2019"},{"key":"10821_CR95","unstructured":"VISSIM (2023) VISSIM website. https:\/\/www.ptvgroup.com\/en-us\/products\/ptv-vissim"},{"key":"10821_CR96","doi-asserted-by":"publisher","unstructured":"Wagner S, Groh K, K\u00fchbeck T, Knoll A (2019) Towards cross-verification and use of simulation in the assessment of automated driving. In: 2019 IEEE Intelligent Vehicles Symposium (IV), pp 1589\u20131596. https:\/\/doi.org\/10.1109\/IVS.2019.8814268","DOI":"10.1109\/IVS.2019.8814268"},{"key":"10821_CR97","unstructured":"Wu P, Jia X, Chen L, Yan J, Li H, Qiao Y (2022) Trajectory-guided control prediction for end-to-end autonomous driving: a simple yet strong baseline. In: Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS \u201922. Curran Associates Inc., Red Hook, NY, USA"},{"key":"10821_CR98","doi-asserted-by":"publisher","unstructured":"Zeller A (2017) Search-based testing and system testing: A marriage in heaven. SBST, pp 49\u201350. https:\/\/doi.org\/10.1109\/SBST.2017.3","DOI":"10.1109\/SBST.2017.3"},{"key":"10821_CR99","doi-asserted-by":"publisher","unstructured":"Zhang M, Zhang Y, Zhang L, Liu C, Khurshid S (2018) DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems. In: Proceedings of the 33rd ACM\/IEEE ASE, ASE \u201918, pp 132\u2013142. ACM. https:\/\/doi.org\/10.1145\/3238147.3238187","DOI":"10.1145\/3238147.3238187"},{"key":"10821_CR100","doi-asserted-by":"publisher","unstructured":"Zhang M, Zhang Y, Zhang L, Liu C, Khurshid S (2018) Deeproad: Gan-based metamorphic testing and input validation framework for autonomous driving systems. In: Proceedings of ASE \u201918, ASE 2018, pp 132\u2013142. ACM, New York, NY, USA. https:\/\/doi.org\/10.1145\/3238147.3238187","DOI":"10.1145\/3238147.3238187"},{"issue":"4","key":"10821_CR101","doi-asserted-by":"publisher","first-page":"1860","DOI":"10.1109\/TSE.2022.3195640","volume":"49","author":"Z Zhong","year":"2023","unstructured":"Zhong Z, Kaiser G, Ray B (2023) Neural network guided evolutionary fuzzing for finding traffic violations of autonomous vehicles. IEEE Trans Softw Eng 49(4):1860\u20131875. https:\/\/doi.org\/10.1109\/TSE.2022.3195640","journal-title":"IEEE Trans Softw Eng"},{"key":"10821_CR102","doi-asserted-by":"publisher","unstructured":"Zohdinasab T, Riccio V, Gambi A, Tonella P (2021) Deephyperion: exploring the feature space of deep learning-based systems through illumination search. In: Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2021, pp 79\u201390. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3460319.3464811","DOI":"10.1145\/3460319.3464811"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-026-10821-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-026-10821-7","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-026-10821-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T11:11:06Z","timestamp":1781608266000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-026-10821-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2,20]]},"references-count":102,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2026,7]]}},"alternative-id":["10821"],"URL":"https:\/\/doi.org\/10.1007\/s10664-026-10821-7","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2,20]]},"assertion":[{"value":"5 March 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 February 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 February 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of Interest"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical Approval"}},{"value":"Not applicable.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Informed Consent"}},{"value":"Clinical trial number: Not applicable.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Clinical Trial Registration"}}],"article-number":"80"}}