{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,21]],"date-time":"2026-04-21T14:47:55Z","timestamp":1776782875903,"version":"3.51.2"},"reference-count":89,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2025,4,9]],"date-time":"2025-04-09T00:00:00Z","timestamp":1744156800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,4,9]],"date-time":"2025-04-09T00:00:00Z","timestamp":1744156800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100004063","name":"Knut och Alice Wallenbergs Stiftelse","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004063","id-type":"DOI","asserted-by":"publisher"}]},{"name":"EPSRC Model-Driven Engineering Network"},{"name":"UKRI Trustworthy Autonomous Systems Node in Verifiability","award":["EP\/V026801\/2"],"award-info":[{"award-number":["EP\/V026801\/2"]}]},{"name":"EPSRC grant on Verified Simulation for Large Quantum Systems","award":["EP\/Y005244\/1"],"award-info":[{"award-number":["EP\/Y005244\/1"]}]},{"DOI":"10.13039\/501100003252","name":"Lund University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100003252","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Autom Softw Eng"],"published-print":{"date-parts":[[2025,11]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>With the emergence of autonomous vehicles comes the requirement of adequate and rigorous testing, particularly in critical scenarios that \u00a0are both challenging and potentially hazardous. Generating synthetic\u00a0simulation-based critical scenarios for testing autonomous vehicles has therefore received considerable interest, yet it is unclear how such scenarios relate to the actual crash or near-crash scenarios \u00a0in the real world. Consequently, their realism is unknown. In this paper, we define realism as the degree of similarity of synthetic critical scenarios to real-world critical scenarios. We propose a methodology to measure realism using two metrics, namely attribute distribution and Euclidean distance. The methodology extracts various attributes from synthetic and realistic critical scenario datasets and performs a set of statistical tests to compare their distributions and distances. As a proof of concept for our methodology, we compare synthetic collision scenarios from DeepScenario against realistic autonomous vehicle collisions collected by the Department of Motor Vehicles in California, to analyse how well DeepScenario synthetic collision scenarios are aligned with real autonomous vehicle collisions recorded in California. We focus on five key attributes that are extractable from both datasets, and analyse the attribution distribution and distance between scenarios in the two datasets. Further, we derive recommendations to improve the realism of synthetic scenarios based on our analysis. Our study of realism provides a framework that can be replicated and extended for other dataset both concerning real-world and synthetically-generated scenarios.<\/jats:p>","DOI":"10.1007\/s10515-025-00499-4","type":"journal-article","created":{"date-parts":[[2025,4,9]],"date-time":"2025-04-09T18:47:41Z","timestamp":1744224461000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Synthetic versus real: an analysis of critical scenarios for autonomous vehicle testing"],"prefix":"10.1007","volume":"32","author":[{"given":"Qunying","family":"Song","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Avner","family":"Bensoussan","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mohammad Reza","family":"Mousavi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2025,4,9]]},"reference":[{"key":"499_CR1","unstructured":"(grva) - new assessment\/test method for automated driving (natm) guidelines for validating automated driving system (ads). Standard, United Nations Economic Commission for Europe (UNECE) (2023). https:\/\/unece.org\/transport\/documents\/2023\/04\/working-documents\/grva-new-assessmenttest-method-automated-driving-natm"},{"issue":"12","key":"499_CR2","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1115\/1.2017-Dec-9","volume":"139","author":"H Abbas","year":"2017","unstructured":"Abbas, H., O\u2019Kelly, M.E., Rodionova, A., Mangharam, R.: A driver\u2019s license test for driverless vehicles. Mech. Eng. 139(12), 13\u201316 (2017). https:\/\/doi.org\/10.1115\/1.2017-Dec-9","journal-title":"Mech. Eng."},{"key":"499_CR3","doi-asserted-by":"publisher","unstructured":"Abdessalem, R.B., Nejati, S., Briand, L.C., Stifter, T.: Testing vision-based control systems using learnable evolutionary algorithms. In: Proceedings of the 40th International Conference on Software Engineering, pp. 1016\u20131026 (2018a). https:\/\/doi.org\/10.1145\/3180155.3180160","DOI":"10.1145\/3180155.3180160"},{"key":"499_CR4","doi-asserted-by":"publisher","unstructured":"Abdessalem, R.B., Panichella, A., Nejati, S., Briand, L.C., Stifter, T.: Testing autonomous cars for feature interaction failures using many-objective search. In: Proceedings of the 33rd ACM\/IEEE International Conference on Automated Software Engineering. ASE \u201918, pp. 143\u2013154. Association for Computing Machinery, New York, NY, USA (2018b). https:\/\/doi.org\/10.1145\/3238147.3238192","DOI":"10.1145\/3238147.3238192"},{"key":"499_CR5","doi-asserted-by":"publisher","unstructured":"Araujo, H., Hoenselaar, T., Mousavi, M.R., Vinel, A.: Connected automated driving: A model-based approach to the analysis of basic awareness services. In: 2020 IEEE 31st Annual International Symposium on Personal, Indoor and Mobile Radio Communications, pp. 1\u20137 (2020). https:\/\/doi.org\/10.1109\/PIMRC48278.2020.9217142","DOI":"10.1109\/PIMRC48278.2020.9217142"},{"key":"499_CR6","doi-asserted-by":"publisher","unstructured":"Bagschik, G., Menzel, T., Maurer, M.: Ontology based scene creation for the development of automated vehicles. In: 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 1813\u20131820 (2018). https:\/\/doi.org\/10.1109\/IVS.2018.8500632","DOI":"10.1109\/IVS.2018.8500632"},{"key":"499_CR7","doi-asserted-by":"publisher","first-page":"374","DOI":"10.1016\/j.trf.2024.04.007","volume":"104","author":"J B\u00e4rgman","year":"2024","unstructured":"B\u00e4rgman, J., Sv\u00e4rd, M., Lundell, S., Hartelius, E.: Methodological challenges of scenario generation validation: a rear-end crash-causation model for virtual safety assessment. Transp. Res. F: Traffic Psychol. Behav. 104, 374\u2013410 (2024)","journal-title":"Transp. Res. F: Traffic Psychol. Behav."},{"key":"499_CR8","doi-asserted-by":"publisher","unstructured":"Beringhoff, F., Greenyer, J., Roesener, C., Tichy, M.: Thirty-one challenges in testing automated vehicles: Interviews with experts from industry and research. In: 2022 IEEE Intelligent Vehicles Symposium (IV), pp. 360\u2013366 (2022). https:\/\/doi.org\/10.1109\/IV51971.2022.9827097 . IEEE","DOI":"10.1109\/IV51971.2022.9827097"},{"key":"499_CR9","doi-asserted-by":"publisher","unstructured":"Birchler, C., Ganz, N., Khatiri, S., Gambi, A., Panichella, S.: Cost-effective simulation-based test selection in self-driving cars software. Sci. Comput. Program. 226, 102926 (2023a) https:\/\/doi.org\/10.1016\/j.scico.2023.102926","DOI":"10.1016\/j.scico.2023.102926"},{"key":"499_CR10","doi-asserted-by":"publisher","unstructured":"Birchler, C., Khatiri, S., Bosshard, B., Gambi, A., Panichella, S.: Machine learning-based test selection for simulation-based testing of self-driving cars software. Empir. Softw. Eng. 28(3), 71 (2023b). https:\/\/doi.org\/10.1007\/s10664-023-10286-y","DOI":"10.1007\/s10664-023-10286-y"},{"key":"499_CR11","doi-asserted-by":"crossref","unstructured":"Braun, T., Fuchs, J., Reisgys, F., Ries, L., Plaum, J., Sch\u00fctt, B., Sax, E.: A review of scenario similarity measures for validation of highly automated driving. In: 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC), pp. 689\u2013696 (2023). IEEE","DOI":"10.1109\/ITSC57777.2023.10422046"},{"key":"499_CR12","unstructured":"Brown, J.: Choosing the right number of components or factors in pca and efa. JALT Test. & Eval. SIG Newslett. 13(2) (2009)"},{"key":"499_CR13","unstructured":"Bryant, F.B., Yarnold, P.R.: Principal-components Analysis and Exploratory and Confirmatory Factor Analysis., pp. 99\u2013136. American Psychological Association, Washington, DC, US (1995)"},{"key":"499_CR14","unstructured":"California collision manual, chapters 1-13, revised 2003. Regulation, United States Department of Transportation (2003). https:\/\/www.nhtsa.gov\/document\/california-collision-manual-chapters-1-13-revised-2003"},{"key":"499_CR15","unstructured":"California Department of Motor Vehicles: Autonomous vehicles collision reports. Available online at: https:\/\/www.dmv.ca.gov\/portal\/vehicle-industry-services\/autonomous-vehicles\/autonomous-vehicle-collision-reports\/ (last accessed: September 1 2023) (2023)"},{"key":"499_CR16","unstructured":"California Department of Motor Vehicles: Autonomous vehicles disengagement reports. Available online at: https:\/\/www.dmv.ca.gov\/portal\/vehicle-industry-services\/autonomous-vehicles\/disengagement-reports\/ (last accessed: September 1 2023) (2022)"},{"key":"499_CR17","doi-asserted-by":"publisher","unstructured":"Cal\u00f2, A., Arcaini, P., Ali, S., Hauer, F., Ishikawa, F.: Generating avoidable collision scenarios for testing autonomous driving systems. In: 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST), pp. 375\u2013386 (2020). https:\/\/doi.org\/10.1109\/ICST46399.2020.00045 . IEEE","DOI":"10.1109\/ICST46399.2020.00045"},{"key":"499_CR18","unstructured":"Commission implementing regulation (eu) 2022\/1426. Regulation, European Parliament and the Council (2022). https:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/?uri=CELEX%3A32022R1426"},{"issue":"1","key":"499_CR19","doi-asserted-by":"publisher","first-page":"5","DOI":"10.23977\/accaf.2020.010102","volume":"1","author":"M Cui","year":"2020","unstructured":"Cui, M.: Introduction to the k-means clustering algorithm based on the elbow method. Account. Audit. Finance 1(1), 5\u20138 (2020). https:\/\/doi.org\/10.23977\/accaf.2020.010102","journal-title":"Account. Audit. Finance"},{"key":"499_CR20","doi-asserted-by":"publisher","DOI":"10.1007\/s10489-022-04148-1","author":"IG Daza","year":"2022","unstructured":"Daza, I.G., Izquierdo, R., Mart\u00ednez, L.M., Benderius, O., Llorca, D.F.: Sim-to-real transfer and reality gap modeling in model predictive control for autonomous driving. Appl. Intell. (2022). https:\/\/doi.org\/10.1007\/s10489-022-04148-1","journal-title":"Appl. Intell."},{"key":"499_CR21","unstructured":"Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: Carla: An open urban driving simulator. In: Conference on Robot Learning, pp. 1\u201316 (2017). PMLR"},{"key":"499_CR22","doi-asserted-by":"publisher","unstructured":"Erdogan, A., Kaplan, E., Leitner, A., Nager, M.: Parametrized end-to-end scenario generation architecture for autonomous vehicles. In: 6th International Conference on Control Engineering & Information Technology (CEIT), pp. 1\u20136 (2018). https:\/\/doi.org\/10.1109\/CEIT.2018.8751872 . IEEE","DOI":"10.1109\/CEIT.2018.8751872"},{"key":"499_CR23","doi-asserted-by":"publisher","unstructured":"Farjo, J., Abou\u00a0Assi, R., Masri, W., Zaraket, F.: Does principal component analysis improve cluster-based analysis? In: 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops, pp. 400\u2013403 (2013). https:\/\/doi.org\/10.1109\/ICSTW.2013.52 . IEEE","DOI":"10.1109\/ICSTW.2013.52"},{"issue":"9","key":"499_CR24","doi-asserted-by":"publisher","first-page":"0184952","DOI":"10.1371\/journal.pone.0184952","volume":"12","author":"FM Favar\u00f2","year":"2017","unstructured":"Favar\u00f2, F.M., Nader, N., Eurich, S.O., Tripp, M., Varadaraju, N.: Examining accident reports involving autonomous vehicles in California. PLoS ONE 12(9), 0184952 (2017). https:\/\/doi.org\/10.1371\/journal.pone.0184952","journal-title":"PLoS ONE"},{"key":"499_CR25","doi-asserted-by":"publisher","unstructured":"Gambi, A., Huynh, T., Fraser, G.: Automatically reconstructing car crashes from police reports for testing self-driving cars. In: 2019 IEEE\/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), pp. 290\u2013291 (2019a). https:\/\/doi.org\/10.1109\/ICSE-Companion.2019.00119","DOI":"10.1109\/ICSE-Companion.2019.00119"},{"key":"499_CR26","doi-asserted-by":"publisher","unstructured":"Gambi, A., Huynh, T., Fraser, G.: Generating effective test cases for self-driving cars from police reports. ESEC\/FSE 2019, pp. 257\u2013267. Association for Computing Machinery, New York, NY, USA (2019b). https:\/\/doi.org\/10.1145\/3338906.3338942","DOI":"10.1145\/3338906.3338942"},{"key":"499_CR27","doi-asserted-by":"publisher","unstructured":"Gambi, A., Nguyen, V., Ahmed, J., Fraser, G.: Generating critical driving scenarios from accident sketches. In: 2022 IEEE International Conference On Artificial Intelligence Testing (AITest), pp. 95\u2013102 (2022). https:\/\/doi.org\/10.1109\/AITest55621.2022.00022","DOI":"10.1109\/AITest55621.2022.00022"},{"key":"499_CR28","doi-asserted-by":"publisher","first-page":"93","DOI":"10.4271\/2018-01-1066","volume":"1","author":"S Hallerbach","year":"2018","unstructured":"Hallerbach, S., Xia, Y., Eberle, U., Koester, F.: Simulation-based identification of critical scenarios for cooperative and automated vehicles. SAE Int. J. Connect. Autom. Veh. 1, 93\u2013106 (2018). https:\/\/doi.org\/10.4271\/2018-01-1066","journal-title":"SAE Int. J. Connect. Autom. Veh."},{"key":"499_CR29","doi-asserted-by":"publisher","unstructured":"Hauer, F., Gerostathopoulos, I., Schmidt, T., Pretschner, A.: Clustering traffic scenarios using mental models as little as possible. In: 2020 IEEE Intelligent Vehicles Symposium (IV), pp. 1007\u20131012 (2020). https:\/\/doi.org\/10.1109\/IV47402.2020.9304636","DOI":"10.1109\/IV47402.2020.9304636"},{"key":"499_CR30","doi-asserted-by":"publisher","unstructured":"Huang, R., Cui, C., Sun, W., Towey, D.: Poster: Is euclidean distance the best distance measurement for adaptive random testing? In: 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST), pp. 406\u2013409 (2020). https:\/\/doi.org\/10.1109\/ICST46399.2020.00049","DOI":"10.1109\/ICST46399.2020.00049"},{"key":"499_CR31","doi-asserted-by":"publisher","unstructured":"Humaira, H., Rasyidah, R.: Determining the appropiate cluster number using elbow method for k-means algorithm. EAI, Padang, Indonesia (2018). https:\/\/doi.org\/10.4108\/eai.24-1-2018.2292388","DOI":"10.4108\/eai.24-1-2018.2292388"},{"key":"499_CR32","doi-asserted-by":"publisher","unstructured":"Ji, P., Li, R., Xue, Y., Dong, Q., Xiao, L., Xue, R.: Perspective, survey and trends: Public driving datasets and toolsets for autonomous driving virtual test. In: 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pp. 264\u2013269 (2021). https:\/\/doi.org\/10.1109\/ITSC48978.2021.9564428 . IEEE","DOI":"10.1109\/ITSC48978.2021.9564428"},{"issue":"2","key":"499_CR33","doi-asserted-by":"publisher","first-page":"171","DOI":"10.1109\/TIV.2018.2886678","volume":"4","author":"Y Kang","year":"2019","unstructured":"Kang, Y., Yin, H., Berger, C.: Test your self-driving algorithm: an overview of publicly available driving datasets and virtual testing environments. IEEE Trans. Intell. Veh. 4(2), 171\u2013185 (2019). https:\/\/doi.org\/10.1109\/TIV.2018.2886678","journal-title":"IEEE Trans. Intell. Veh."},{"key":"499_CR34","doi-asserted-by":"publisher","unstructured":"Klischat, M., Althoff, M.: Generating critical test scenarios for automated vehicles with evolutionary algorithms. In: 2019 IEEE Intelligent Vehicles Symposium (IV), pp. 2352\u20132358 (2019). https:\/\/doi.org\/10.1109\/IVS.2019.8814230","DOI":"10.1109\/IVS.2019.8814230"},{"key":"499_CR35","doi-asserted-by":"publisher","unstructured":"Knauss, A., Schr\u00f6der, J., Berger, C., Eriksson, H.: Paving the roadway for safety of automated vehicles: an empirical study on testing challenges. In: 2017 IEEE Intelligent Vehicles Symposium (IV), pp. 1873\u20131880 (2017). https:\/\/doi.org\/10.1109\/IVS.2017.7995978 . IEEE","DOI":"10.1109\/IVS.2017.7995978"},{"key":"499_CR36","doi-asserted-by":"publisher","unstructured":"Lago, P., Runeson, P., Song, Q., Verdecchia, R.: Threats to validity in software engineering - hypocritical paper section or essential analysis? In: Proceedings of the 18th ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement. ESEM \u201924, pp. 314\u2013324. Association for Computing Machinery, New York, NY, USA (2024). https:\/\/doi.org\/10.1145\/3674805.3686691","DOI":"10.1145\/3674805.3686691"},{"key":"499_CR37","doi-asserted-by":"publisher","unstructured":"Li, G., Li, Y., Jha, S., Tsai, T., Sullivan, M., Hari, S.K.S., Kalbarczyk, Z., Iyer, R.: Av-fuzzer: Finding safety violations in autonomous driving systems. In: 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE), pp. 25\u201336 (2020). https:\/\/doi.org\/10.1109\/ISSRE5003.2020.00012 . IEEE","DOI":"10.1109\/ISSRE5003.2020.00012"},{"key":"499_CR38","doi-asserted-by":"publisher","unstructured":"Li, S., Wang, W., Mo, Z., Zhao, D.: Cluster naturalistic driving encounters using deep unsupervised learning. In: 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 1354\u20131359 (2018). https:\/\/doi.org\/10.1109\/IVS.2018.8500529","DOI":"10.1109\/IVS.2018.8500529"},{"issue":"5","key":"499_CR39","doi-asserted-by":"publisher","first-page":"986","DOI":"10.1109\/TFUZZ.2020.2966182","volume":"29","author":"F Liu","year":"2021","unstructured":"Liu, F., Deng, Y.: Determine the number of unknown targets in open world based on elbow method. IEEE Trans. Fuzzy Syst. 29(5), 986\u2013995 (2021). https:\/\/doi.org\/10.1109\/TFUZZ.2020.2966182","journal-title":"IEEE Trans. Fuzzy Syst."},{"key":"499_CR40","doi-asserted-by":"publisher","unstructured":"Lou, G., Deng, Y., Zheng, X., Zhang, M., Zhang, T.: Testing of autonomous driving systems: where are we and where should we go? In: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ESEC\/FSE 2022, pp. 31\u201343. Association for Computing Machinery, New York, NY, USA (2022). https:\/\/doi.org\/10.1145\/3540250.3549111","DOI":"10.1145\/3540250.3549111"},{"key":"499_CR41","doi-asserted-by":"publisher","unstructured":"Lu, C., Yue, T., Ali, S.: Deepscenario: An open driving scenario dataset for autonomous driving system testing. In: 2023 IEEE\/ACM 20th International Conference on Mining Software Repositories (MSR), pp. 52\u201356 (2023a). https:\/\/doi.org\/10.1109\/MSR59073.2023.00020","DOI":"10.1109\/MSR59073.2023.00020"},{"key":"499_CR42","doi-asserted-by":"publisher","unstructured":"Lu, C., Yue, T., Ali, S.: DeepScenario: an open driving scenario dataset for autonomous driving system testing. Zenodo (2023b). https:\/\/doi.org\/10.5281\/zenodo.7714194","DOI":"10.5281\/zenodo.7714194"},{"key":"499_CR43","doi-asserted-by":"publisher","unstructured":"Lu, C., Shi, Y., Zhang, H., Zhang, M., Wang, T., Yue, T., Ali, S.: Learning configurations of operating environment of autonomous vehicles to maximize their collisions. IEEE Trans. Softw. Eng. 49(1), 384\u2013402 (2023c). https:\/\/doi.org\/10.1109\/TSE.2022.3150788","DOI":"10.1109\/TSE.2022.3150788"},{"key":"499_CR44","doi-asserted-by":"publisher","unstructured":"Luo, Y., Zhang, X.-Y., Arcaini, P., Jin, Z., Zhao, H., Ishikawa, F., Wu, R., Xie, T.: Targeting requirements violations of autonomous driving systems by dynamic evolutionary search. In: 2021 36th IEEE\/ACM International Conference on Automated Software Engineering (ASE), pp. 279\u2013291 (2021). https:\/\/doi.org\/10.1109\/ASE51524.2021.9678883 . IEEE","DOI":"10.1109\/ASE51524.2021.9678883"},{"key":"499_CR45","unstructured":"National Highway Traffic Safety Administration: Standing General Order on Crash Reporting - Data. Available online: https:\/\/www.nhtsa.gov\/laws-regulations\/standing-general-order-crash-reporting (Accessed on Sept 01 2023) (2023)"},{"key":"499_CR46","unstructured":"National Transportation Safety Board: Automated Vehicles - Investigations. Available online: https:\/\/www.ntsb.gov\/Advocacy\/safety-topics\/Pages\/automated-vehicles-investigations.aspx (Accessed on Sept 01 2023) (2018)"},{"key":"499_CR47","doi-asserted-by":"crossref","unstructured":"Neelofar, N., Aleti, A.: Towards reliable ai: Adequacy metrics for ensuring the quality of system-level testing of autonomous vehicles. In: Proceedings of the 46th IEEE\/ACM International Conference on Software Engineering, pp. 1\u201312 (2024)","DOI":"10.1145\/3597503.3623314"},{"key":"499_CR48","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2023.107286","volume":"163","author":"S Nejati","year":"2023","unstructured":"Nejati, S., Sorokin, L., Safin, D., Formica, F., Mahboob, M.M., Menghi, C.: Reflections on surrogate-assisted search-based testing: a taxonomy and two replication studies based on industrial ADAS and simulink models. Inf. Softw. Technol. 163, 107286 (2023). https:\/\/doi.org\/10.1016\/j.infsof.2023.107286","journal-title":"Inf. Softw. Technol."},{"key":"499_CR49","doi-asserted-by":"crossref","unstructured":"Neurohr, B., Graaff, T., Eggers, A., Bienm\u00fcller, T., M\u00f6hlmann, E.: Providing evidence for the validity of the virtual verification of automated driving systems. In: European Dependable Computing Conference, pp. 5\u201313 (2024). Springer","DOI":"10.1007\/978-3-031-56776-6_1"},{"key":"499_CR50","doi-asserted-by":"publisher","unstructured":"Neurohr, C., Westhofen, L., Henning, T., Graaff, T., M\u00f6hlmann, E., B\u00f6de, E.: Fundamental considerations around scenario-based testing for automated driving. In: 2020 IEEE Intelligent Vehicles Symposium (IV), pp. 121\u2013127 (2020). https:\/\/doi.org\/10.1109\/IV47402.2020.9304823","DOI":"10.1109\/IV47402.2020.9304823"},{"issue":"1","key":"499_CR51","doi-asserted-by":"publisher","first-page":"149","DOI":"10.1016\/j.aap.2010.12.006","volume":"44","author":"D Otte","year":"2012","unstructured":"Otte, D., J\u00e4nsch, M., Haasper, C.: Injury protection and accident causation parameters for vulnerable road users based on German in-depth accident study gidas. Accid. Anal. Prev. 44(1), 149\u2013153 (2012)","journal-title":"Accid. Anal. Prev."},{"key":"499_CR52","doi-asserted-by":"publisher","first-page":"150","DOI":"10.1016\/j.ssci.2019.03.010","volume":"119","author":"E Papazikou","year":"2019","unstructured":"Papazikou, E., Quddus, M., Thomas, P., Kidd, D.: What came before the crash? an investigation through shrp2 nds data. Saf. Sci. 119, 150\u2013161 (2019)","journal-title":"Saf. Sci."},{"key":"499_CR53","doi-asserted-by":"publisher","unstructured":"Pearl, J.: Causality: Models, Reasoning, and Inference. Cambridge University Press, Cambridge, UK (2009). https:\/\/doi.org\/10.1017\/S0266466603004109","DOI":"10.1017\/S0266466603004109"},{"key":"499_CR54","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1016\/j.trpro.2020.03.003","volume":"45","author":"D Petrovic","year":"2020","unstructured":"Petrovic, D., Mijailovic, R., Pesic, D.: Traffic accidents with autonomous vehicles: type of collisions, manoeuvres and errors of conventional vehicles\u2019 drivers. Transp. Res. Procedia 45, 161\u2013168 (2020). https:\/\/doi.org\/10.1016\/j.trpro.2020.03.003","journal-title":"Transp. Res. Procedia"},{"key":"499_CR55","doi-asserted-by":"publisher","unstructured":"Pokorny, P., H\u00f8ye, A.: Descriptive analysis of reports on autonomous vehicle collisions in california: January 2021-june 2022. Traffic Saf. Res. 2 (2022) https:\/\/doi.org\/10.55329\/xydm4000","DOI":"10.55329\/xydm4000"},{"key":"499_CR56","unstructured":"PyPI: PyPDF2 3.0.1. Available online at: https:\/\/pypi.org\/project\/PyPDF2\/ (Accessed on June 2 2023) (2022)"},{"key":"499_CR57","unstructured":"Reisgys, F., Elgharbawy, M., Schwarzhaupt, A., Sax, E., Kemeny, A., Chardonnet, J., Colombet, F.: Argumentation on adas simulation validity using aleatory and epistemic uncertainty estimation. In: Proceedings of the Driving Simulation Conference 2021 Europe VR, pp. 25\u201332 (2021). Munich, Germany"},{"key":"499_CR58","doi-asserted-by":"publisher","first-page":"87456","DOI":"10.1109\/ACCESS.2020.2993730","volume":"8","author":"S Riedmaier","year":"2020","unstructured":"Riedmaier, S., Ponn, T., Ludwig, D., Schick, B., Diermeyer, F.: Survey on scenario-based safety assessment of automated vehicles. IEEE Access 8, 87456\u201387477 (2020). https:\/\/doi.org\/10.1109\/ACCESS.2020.2993730","journal-title":"IEEE Access"},{"key":"499_CR59","doi-asserted-by":"publisher","first-page":"2655","DOI":"10.1007\/s11831-020-09473-7","volume":"28","author":"S Riedmaier","year":"2021","unstructured":"Riedmaier, S., Danquah, B., Schick, B., Diermeyer, F.: Unified framework and survey for model verification, validation and uncertainty quantification. Archiv. Comput. Methods Eng. 28, 2655\u20132688 (2021)","journal-title":"Archiv. Comput. Methods Eng."},{"key":"499_CR60","doi-asserted-by":"publisher","DOI":"10.1016\/j.simpat.2021.102274","volume":"109","author":"S Riedmaier","year":"2021","unstructured":"Riedmaier, S., Schneider, J., Danquah, B., Schick, B., Diermeyer, F.: Non-deterministic model validation methodology for simulation-based safety assessment of automated vehicles. Simul. Model. Pract. Theory 109, 102274 (2021)","journal-title":"Simul. Model. Pract. Theory"},{"key":"499_CR61","unstructured":"Road vehicles - safety of the intended functionality. Standard, International Organization for Standardization (2022). https:\/\/www.iso.org\/obp\/ui\/#iso:std:77490:en"},{"issue":"3","key":"499_CR62","doi-asserted-by":"publisher","first-page":"648","DOI":"10.3390\/s19030648","volume":"19","author":"F Rosique","year":"2019","unstructured":"Rosique, F., Navarro, P.J., Fern\u00e1ndez, C., Padilla, A.: A systematic review of perception system and simulators for autonomous vehicles research. Sensors 19(3), 648 (2019). https:\/\/doi.org\/10.3390\/s19030648","journal-title":"Sensors"},{"key":"499_CR63","doi-asserted-by":"crossref","unstructured":"Sargent, R.G.: Verification and validation of simulation models. In: Proceedings of the 2010 Winter Simulation Conference, pp. 166\u2013183 (2010). IEEE","DOI":"10.1109\/WSC.2010.5679166"},{"key":"499_CR64","unstructured":"Scenario-based verification and validation of self-driving vehicles: relevant safety metrics. White paper, Siemens Digital Industries Software (2022)"},{"key":"499_CR65","doi-asserted-by":"publisher","first-page":"59131","DOI":"10.1109\/ACCESS.2021.3072739","volume":"9","author":"M Scholtes","year":"2021","unstructured":"Scholtes, M., Westhofen, L., Turner, L.R., Lotto, K., Schuldes, M., Weber, H., Wagener, N., Neurohr, C., Bollmann, M.H., K\u00f6rtke, F., et al.: 6-layer model for a structured description and categorization of urban traffic and environment. IEEE Access 9, 59131\u201359147 (2021). https:\/\/doi.org\/10.1109\/ACCESS.2021.3072739","journal-title":"IEEE Access"},{"issue":"1","key":"499_CR66","doi-asserted-by":"publisher","first-page":"298","DOI":"10.1038\/s41597-021-01083-7","volume":"8","author":"A Sinha","year":"2021","unstructured":"Sinha, A., Chand, S., Vu, V., Chen, H., Dixit, V.: Crash and disengagement data of autonomous vehicles on public roads in California. Sci. Data 8(1), 298 (2021). https:\/\/doi.org\/10.1038\/s41597-021-01083-7","journal-title":"Sci. Data"},{"key":"499_CR67","doi-asserted-by":"crossref","unstructured":"Song, Q., Engstr\u00f6m, E., Runeson, P.: An empirically grounded path forward for scenario-based testing of autonomous driving systems. In: Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, pp. 232\u2013243 (2024a)","DOI":"10.1145\/3663529.3663843"},{"key":"499_CR68","doi-asserted-by":"publisher","unstructured":"Song, Q., Engstr\u00f6m, E., Runeson, P.: Industry practices for challenging autonomous driving systems with critical scenarios. ACM Trans. Softw. Eng. Methodol. 33(4), 1\u201335 (2024b). https:\/\/doi.org\/10.1145\/3640334","DOI":"10.1145\/3640334"},{"issue":"2","key":"499_CR69","doi-asserted-by":"publisher","first-page":"441","DOI":"10.1007\/s11219-022-09604-2","volume":"31","author":"Q Song","year":"2023","unstructured":"Song, Q., Tan, K., Runeson, P., Persson, S.: Critical scenario identification for realistic testing of autonomous driving systems. Softw. Qual. J. 31(2), 441\u2013469 (2023). https:\/\/doi.org\/10.1007\/s11219-022-09604-2","journal-title":"Softw. Qual. J."},{"key":"499_CR70","doi-asserted-by":"publisher","unstructured":"Song, Q., Avner, B., Mohammad, R.M.: ADT-CSE. Zenodo (2024c). https:\/\/doi.org\/10.5281\/zenodo.11179407","DOI":"10.5281\/zenodo.11179407"},{"key":"499_CR71","unstructured":"spicy: scipy.stats.chisquare. Available online at: https:\/\/docs.scipy.org\/doc\/scipy\/reference\/generated\/scipy.stats.chisquare.html (Accessed on June 2 2023) (2023)"},{"key":"499_CR72","doi-asserted-by":"publisher","unstructured":"Stocco, A., Pulfer, B., Tonella, P.: Mind the gap! a study on the transferability of virtual versus physical-world testing of autonomous driving systems. IEEE Trans. Softw. Eng. 49(4), 1928\u20131940 (2023a). https:\/\/doi.org\/10.1109\/TSE.2022.3202311","DOI":"10.1109\/TSE.2022.3202311"},{"key":"499_CR73","doi-asserted-by":"publisher","unstructured":"Stocco, A., Pulfer, B., Tonella, P.: Model vs system level testing of autonomous driving systems: a replication and extension study. Empir. Softw. Eng. 28(3), 73 (2023b). https:\/\/doi.org\/10.1007\/s10664-023-10306-x","DOI":"10.1007\/s10664-023-10306-x"},{"issue":"9","key":"499_CR74","doi-asserted-by":"publisher","first-page":"14088","DOI":"10.1109\/TITS.2021.3136353","volume":"23","author":"J Sun","year":"2021","unstructured":"Sun, J., Zhang, H., Zhou, H., Yu, R., Tian, Y.: Scenario-based test automation for highly automated vehicles: a review and paving the way for systematic safety assurance. IEEE Trans. Intell. Transp. Syst. 23(9), 14088\u201314103 (2021). https:\/\/doi.org\/10.1109\/TITS.2021.3136353","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"499_CR75","doi-asserted-by":"publisher","DOI":"10.1145\/3579642","author":"S Tang","year":"2023","unstructured":"Tang, S., Zhang, Z., Zhang, Y., Zhou, J., Guo, Y., Liu, S., Guo, S., Li, Y.-F., Ma, L., Xue, Y., et al.: A survey on automated driving system testing: landscapes and trends. ACM Trans. Softw. Eng. Methodol. (2023). https:\/\/doi.org\/10.1145\/3579642","journal-title":"ACM Trans. Softw. Eng. Methodol."},{"key":"499_CR76","doi-asserted-by":"publisher","unstructured":"Tenbrock, A., K\u00f6nig, A., Keutgens, T., Weber, H.: The conscend dataset: concrete scenarios from the highd dataset according to alks regulation unece r157 in openx. In: 2021 IEEE Intelligent Vehicles Symposium Workshops (IV Workshops), pp. 174\u2013181 (2021). https:\/\/doi.org\/10.1109\/IVWorkshops54471.2021.9669219","DOI":"10.1109\/IVWorkshops54471.2021.9669219"},{"key":"499_CR77","unstructured":"The SVL Simulator team: SVL Simulator: An Autonomous Vehicle Simulator. Available online: https:\/\/github.com\/lgsvl\/simulator (Accessed on Aug 21 2023) (2020)"},{"key":"499_CR78","unstructured":"timeanddate: San Francisco, California, USA - Sunrise, Sunset, and Daylength, September 2023. Available online at: https:https:\/\/www.timeanddate.com\/sun\/usa\/san-francisco (last accessed: June 2 2023) (2023)"},{"key":"499_CR79","unstructured":"United States Department of Transportation: FARS EncyclopediaL Help - Terms. Available online: https:\/\/www-fars.nhtsa.dot.gov\/help\/terms.aspx (Accessed on Aug 21 2023) (2023)"},{"key":"499_CR80","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2023.107329","volume":"164","author":"R Verdecchia","year":"2023","unstructured":"Verdecchia, R., Engstr\u00f6m, E., Lago, P., Runeson, P., Song, Q.: Threats to validity in software engineering research: a critical reflection. Inf. Softw. Technol. 164, 107329 (2023). https:\/\/doi.org\/10.1016\/j.infsof.2023.107329","journal-title":"Inf. Softw. Technol."},{"issue":"sup1","key":"499_CR81","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1080\/15389588.2019.1616086","volume":"20","author":"P Wimmer","year":"2019","unstructured":"Wimmer, P., D\u00fcring, M., Chajmowicz, H., Granum, F., King, J., Kolk, H., Camp, O., Scognamiglio, P., Wagner, M.: Toward harmonizing prospective effectiveness assessment for road safety: comparing tools in standard test case simulations. Traffic Inj. Prev. 20(sup1), 139\u2013145 (2019)","journal-title":"Traffic Inj. Prev."},{"key":"499_CR82","unstructured":"WorldClimate.com: Average Weather Data for San Diego, California. Available online: http:\/\/www.worldclimate.com\/climate\/us\/california\/san-diego (Accessed on Aug 21 2023) (2023)"},{"key":"499_CR83","unstructured":"WorldClimate.com: Average Weather Data for San Francisco, California. Available online: http:\/\/www.worldclimate.com\/climate\/us\/california\/san-francisco (Accessed on Aug 21 2023) (2023)"},{"key":"499_CR84","doi-asserted-by":"publisher","unstructured":"Xinxin, Z., Fei, L., Xiangbin, W.: Csg: Critical scenario generation from real traffic accidents. In: 2020 IEEE Intelligent Vehicles Symposium (IV), pp. 1330\u20131336 (2020). https:\/\/doi.org\/10.1109\/IV47402.2020.9304609","DOI":"10.1109\/IV47402.2020.9304609"},{"issue":"1","key":"499_CR85","doi-asserted-by":"publisher","first-page":"2037","DOI":"10.1038\/s41467-023-37677-5","volume":"14","author":"X Yan","year":"2023","unstructured":"Yan, X., Zou, Z., Feng, S., Zhu, H., Sun, H., Liu, H.X.: Learning naturalistic driving environment with statistical realism. Nat. Commun. 14(1), 2037 (2023)","journal-title":"Nat. Commun."},{"key":"499_CR86","unstructured":"Yuqi Huai: SORA-SVL: Local Cloud built for SVL Simulator. Available online: https:\/\/github.com\/YuqiHuai\/SORA-SVL (Accessed on Aug 21 2023) (2022)"},{"issue":"3","key":"499_CR87","doi-asserted-by":"publisher","first-page":"991","DOI":"10.1109\/TSE.2022.3170122","volume":"49","author":"X Zhang","year":"2023","unstructured":"Zhang, X., Tao, J., Tan, K., T\u00f6rngren, M., S\u00e1nchez, J.M.G., Ramli, M.R., Tao, X., Gyllenhammar, M., Wotawa, F., Mohan, N., Nica, M., Felbinger, H.: Finding critical scenarios for automated driving systems: a systematic mapping study. IEEE Trans. Software Eng. 49(3), 991\u20131026 (2023). https:\/\/doi.org\/10.1109\/TSE.2022.3170122","journal-title":"IEEE Trans. Software Eng."},{"key":"499_CR88","unstructured":"Zhong, Z., Tang, Y., Zhou, Y., Neves, V.d.O., Liu, Y., Ray, B.: A survey on scenario-based testing for automated driving systems in high-fidelity simulation. arXiv preprint arXiv:2112.00964 (2021)"},{"key":"499_CR89","doi-asserted-by":"publisher","unstructured":"Zhou, Y.: A review of text classification based on deep learning. In: Proceedings of the 2020 3rd International Conference on Geoinformatics and Data Analysis, pp. 132\u2013136. Association for Computing Machinery, ??? (2020). https:\/\/doi.org\/10.1145\/3397056.3397082","DOI":"10.1145\/3397056.3397082"}],"container-title":["Automated Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10515-025-00499-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10515-025-00499-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10515-025-00499-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,10]],"date-time":"2025-09-10T13:58:14Z","timestamp":1757512694000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10515-025-00499-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4,9]]},"references-count":89,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2025,11]]}},"alternative-id":["499"],"URL":"https:\/\/doi.org\/10.1007\/s10515-025-00499-4","relation":{},"ISSN":["0928-8910","1573-7535"],"issn-type":[{"value":"0928-8910","type":"print"},{"value":"1573-7535","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,4,9]]},"assertion":[{"value":"11 May 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 February 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 April 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"There is no financial or non-financial interests that are directly or indirectly related to the work submitted for publication.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"37"}}