{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,20]],"date-time":"2026-05-20T01:01:14Z","timestamp":1779238874169,"version":"3.51.4"},"reference-count":76,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2025,2,23]],"date-time":"2025-02-23T00:00:00Z","timestamp":1740268800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000923","name":"Australian Research Council","doi-asserted-by":"crossref","award":["DP210100041"],"award-info":[{"award-number":["DP210100041"]}],"id":[{"id":"10.13039\/501100000923","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2025,3,31]]},"abstract":"<jats:p>\n            Before being deployed on roads, Autonomous Vehicles (AVs) must undergo comprehensive testing. Safety-critical situations, however, are infrequent in usual driving conditions, so simulated scenarios are used to create them. A test scenario comprises static and dynamic features related to the AV and the test environment; the representation of these features is complex and makes testing a heavy process. A test scenario is effective if it identifies incorrect behaviors of the AV. In this article, we present a technique for identifying key features of test scenarios associated with their effectiveness using Instance Space Analysis (ISA). ISA generates a (\n            <jats:inline-formula content-type=\"math\/tex\">\n              <jats:tex-math notation=\"LaTeX\" version=\"MathJax\">\\(2D\\)<\/jats:tex-math>\n            <\/jats:inline-formula>\n            ) representation of test scenarios and their features. This visualization helps to identify combinations of features that make a test scenario effective. We present a graphical representation of each feature that helps identify how well each testing technique explores the search space. While identifying key features is a primary goal, this study specifically seeks to determine the critical features that differentiate the performance of algorithms. Finally, we present metrics to assess the robustness of testing algorithms and the scenarios generated. Collecting essential features in combination with their values associated with effectiveness can be used for selection and prioritization of effective test cases.\n          <\/jats:p>","DOI":"10.1145\/3699596","type":"journal-article","created":{"date-parts":[[2024,10,8]],"date-time":"2024-10-08T15:39:02Z","timestamp":1728401942000},"page":"1-36","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Instance Space Analysis of Testing of Autonomous Vehicles in Critical Scenarios"],"prefix":"10.1145","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2783-098X","authenticated-orcid":false,"given":"Victor","family":"Crespo-Rodriguez","sequence":"first","affiliation":[{"name":"Faculty of Information Technology, Monash University, Clayton, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2572-0250","authenticated-orcid":false,"family":"Neelofar","sequence":"additional","affiliation":[{"name":"Faculty of Information Technology, Monash University, Clayton, Australia and RMIT University, Melbourne, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1716-690X","authenticated-orcid":false,"given":"Aldeida","family":"Aleti","sequence":"additional","affiliation":[{"name":"Faculty of Information Technology, Monash University, Clayton, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1511-2163","authenticated-orcid":false,"given":"Burak","family":"Turhan","sequence":"additional","affiliation":[{"name":"Faculty of ITEE, University of Oulu, Oulu, Finland and Monash University, Clayton, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,2,23]]},"reference":[{"key":"e_1_3_1_2_2","unstructured":"BeamNG.tech. Retrieved from https:\/\/beamng.tech\/"},{"key":"e_1_3_1_3_2","unstructured":"GitHub. ApolloAuto\/apollo: An open autonomous driving platform. Retrieved July 1 2024 from https:\/\/github.com\/ApolloAuto\/apollo"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238192"},{"key":"e_1_3_1_5_2","unstructured":"National Highway Traffic Safety Administration. 2022. Summary Report: Standing General Order on Crash Reporting for Automated Driving Systems. 9 pages. Retrieved from https:\/\/www.nhtsa.gov\/sites\/nhtsa.gov\/files\/2022-06\/ADS-SGO-Report-June-2022.pdf"},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3243522"},{"key":"e_1_3_1_7_2","doi-asserted-by":"crossref","unstructured":"Aldeida Aleti and Matias Martinez. 2021. E-APR: Mapping the effectiveness of automated program repair techniques. Empirical Software Engineering 26 (2021) 1\u201330.","DOI":"10.1007\/s10664-021-09989-x"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejor.2022.04.012"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCIMA.2007.328"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST49551.2021.00042"},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2018.09.055"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/IVS.2018.8500632"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER53432.2022.00030"},{"key":"e_1_3_1_14_2","unstructured":"Christian Birchler Sajad Khatiri Pouria Derakhshanfar Sebastiano Panichella and Annibale Panichella. 2021. Automated test cases prioritization for self-driving cars in virtual environments. arXiv:2107.09614. Retrieved from https:\/\/arxiv.org\/abs\/2107.09614"},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533818"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/QRS54544.2021.00028"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/SBST52555.2021.00016"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2021.3091477"},{"key":"e_1_3_1_19_2","unstructured":"Junjie Chen Ming Yan Zan Wang Yuning Kang and Zhuo Wu. 2020. Deep neural network test coverage: How far are we? arXiv:2010.04946."},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3644032.3644457"},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1037\/met0000079"},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549152"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2023.3259322"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.1983.1056714"},{"key":"e_1_3_1_25_2","first-page":"226","article-title":"A density-based algorithm for discovering clusters in large spatial databases with noise","volume":"96","author":"Ester Martin","year":"1996","unstructured":"Martin Ester, Hans-Peter Kriegel, J\u00f6rg Sander, and Xiaowei Xu. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining (KDD \u201996), Vol. 96, 226\u2013231.","journal-title":"Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining (KDD \u201996)"},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0184952"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3526072.3527538"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-Companion.2019.00030"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380391"},{"key":"e_1_3_1_30_2","unstructured":"Aur\u00e9lien G\u00e9ron. 2022. Hands-on Machine Learning with Scikit-Learn Keras and TensorFlow. O\u2019Reilly Media Inc."},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/IV48863.2021.9576023"},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/IV47402.2020.9304636"},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/ITSC.2019.8917326"},{"key":"e_1_3_1_34_2","unstructured":"Zhisheng Hu Shengjian Guo Zhenyu Zhong and Kang Li. 2021. Coverage-based scene fuzzing for virtual autonomous driving testing. arXiv:2106.00873. Retrieved from https:\/\/arxiv.org\/abs\/2106.00873"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/SBST52555.2021.00019"},{"key":"e_1_3_1_36_2","unstructured":"Bernhard Kaiser. 2021. Application Story of ODD as part of Safety Assurance."},{"key":"e_1_3_1_37_2","doi-asserted-by":"crossref","unstructured":"Nidhi Kalra and Susan M. Paddock. 2016. Driving to Safety: How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability? RAND Corporation 1\u201316. Retrieved from http:\/\/www.jstor.org\/stable\/10.7249\/j.ctt1btc0xw.1","DOI":"10.7249\/RR1478"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijforecast.2016.09.004"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/LES.2016.2644619"},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00108"},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.01426"},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3550270"},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE5003.2020.00012"},{"key":"e_1_3_1_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3150788"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-88106-1_4"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","unstructured":"Kang Meng Rui Zhou Zhiheng Li and Kai Zhang. 2023. A quantitative approach of generating challenging testing scenarios based on functional safety standard. Applied Sciences 13 6 (Mar. 2023) 3494. DOI: 10.3390\/app13063494","DOI":"10.3390\/app13063494"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0001-4575(00)00019-1"},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/SBST52555.2021.00018"},{"key":"e_1_3_1_50_2","unstructured":"Jean-Baptiste Mouret and Jeff Clune. 2015. Illuminating search spaces by mapping elites. arXiv:1504.04909. Retrieved from http:\/\/arxiv.org\/abs\/1504.04909"},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3436893"},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-017-5629-5"},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","DOI":"10.1162\/evco_a_00194"},{"key":"e_1_3_1_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3640335"},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3623314"},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3228334"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1109\/TR.2018.2832072"},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","DOI":"10.1109\/SBST52555.2021.00011"},{"key":"e_1_3_1_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_3_1_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/ITSC45102.2020.9294422"},{"key":"e_1_3_1_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3068335"},{"key":"e_1_3_1_62_2","article-title":"Instance space analysis for algorithm testing: Methodology and software tools","author":"Smith-Miles Kate","year":"2021","unstructured":"Kate Smith-Miles and Mario Andr\u00e9s Mu\u00f1oz. 2021. Instance space analysis for algorithm testing: Methodology and software tools. Computing Surveys (2021).","journal-title":"Computing Surveys"},{"key":"e_1_3_1_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/CEC.2012.6252992"},{"key":"e_1_3_1_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3358233"},{"key":"e_1_3_1_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/IV48863.2021.9575536"},{"key":"e_1_3_1_66_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA48506.2021.9560890"},{"key":"e_1_3_1_67_2","doi-asserted-by":"publisher","unstructured":"Haoxiang Tian Yan Jiang Guoquan Wu Jiren Yan Jun Wei Wei Chen Shuo Li and Dan Ye. 2022. MOSAT: Finding safety violations of autonomous driving systems using multi-objective genetic algorithm (ESEC\/FSE 2022). ACM New York NY 94\u2013106. DOI: 10.1145\/3540250.3549100","DOI":"10.1145\/3540250.3549100"},{"key":"e_1_3_1_68_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00155"},{"key":"e_1_3_1_69_2","doi-asserted-by":"publisher","DOI":"10.1109\/ITSC.2015.164"},{"key":"e_1_3_1_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/3526072.3527530"},{"key":"e_1_3_1_71_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.aap.2022.106776"},{"key":"e_1_3_1_72_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238187"},{"key":"e_1_3_1_73_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534223"},{"key":"e_1_3_1_74_2","unstructured":"Ziyuan Zhong Yun Tang Yuan Zhou V\u00e2nia de Oliveira Neves Yang Liu and Baishakhi Ray. 2021. A survey on scenario-based testing for automated driving systems in high-fidelity simulation. arXiv:2112.00964. Retrieved from https:\/\/arxiv.org\/abs\/2112.00964"},{"key":"e_1_3_1_75_2","doi-asserted-by":"publisher","DOI":"10.1109\/IVS.2016.7535378"},{"key":"e_1_3_1_76_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464811"},{"key":"e_1_3_1_77_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597926.3598109"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3699596","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3699596","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:09:52Z","timestamp":1750295392000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3699596"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,23]]},"references-count":76,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,3,31]]}},"alternative-id":["10.1145\/3699596"],"URL":"https:\/\/doi.org\/10.1145\/3699596","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,2,23]]},"assertion":[{"value":"2023-08-23","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-09-14","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-02-23","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}