{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T17:03:22Z","timestamp":1783530202014,"version":"3.55.0"},"reference-count":91,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2024,4,18]],"date-time":"2024-04-18T00:00:00Z","timestamp":1713398400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Wallenberg AI, Autonomous Systems and Software Program"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,5,31]]},"abstract":"<jats:p>Testing autonomous driving systems for safety and reliability is essential, yet complex. A primary challenge is identifying relevant test scenarios, especially the critical ones that may expose hazards or harm to autonomous vehicles and other road users. Although numerous approaches and tools for critical scenario identification are proposed, the industry practices for selection, implementation, and evaluation of approaches, are not well understood. Therefore, we aim at exploring practical aspects of how autonomous driving systems are tested, particularly the identification and use of critical scenarios. We interviewed 13 practitioners from 7 companies in autonomous driving in Sweden. We used thematic modeling to analyse and synthesize the interview data. As a result, we present 9 themes of practices and 4 themes of challenges related to critical scenarios. Our analysis indicates there is little joint effort in the industry, despite every approach has its own limitations, and tools and platforms are lacking. To that end, we recommend the industry and academia combine different approaches, collaborate among different stakeholders, and continuously learn the field. The contributions of our study are exploration and synthesis of industry practices and related challenges for critical scenario identification and testing, and potential increase of industry relevance for future studies.<\/jats:p>","DOI":"10.1145\/3640334","type":"journal-article","created":{"date-parts":[[2024,1,11]],"date-time":"2024-01-11T12:26:34Z","timestamp":1704975994000},"page":"1-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":16,"title":["Industry Practices for Challenging Autonomous Driving Systems with Critical Scenarios"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8653-0250","authenticated-orcid":false,"given":"Qunying","family":"Song","sequence":"first","affiliation":[{"name":"Department of Computer Science, Lund University, Lund, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6736-9425","authenticated-orcid":false,"given":"Emelie","family":"Engstr\u00f6m","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Lund University, Lund, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2795-4851","authenticated-orcid":false,"given":"Per","family":"Runeson","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Lund University, Lund, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,4,18]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1115\/1.2017-Dec-9"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180160"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238192"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2021.3131990"},{"key":"e_1_3_3_6_2","unstructured":"apollo. 2023. Apollo Open Platform. Retrieved from https:\/\/developer.apollo.auto\/"},{"key":"e_1_3_3_7_2","unstructured":"apollo. 2023. ApolloAuto\/apollo: An open autonomous driving platform. Retrieved from https:\/\/github.com\/ApolloAuto\/apollo"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICE.2017.8279890"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-021-10072-8"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/2970276.2970311"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/IV51971.2022.9827097"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2018.2841339"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER53432.2022.00030"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-023-10286-y"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533818"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2017.3571562"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.3390\/machines10111101"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST46399.2020.00045"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/ESEM.2011.36"},{"key":"e_1_3_3_20_2","unstructured":"Erwin de Gelder and Olaf Op den Camp. 2020. Procedure for the Safety Assessment of an Autonomous Vehicle Using Real-World Scenarios. CoRR abs\/2012.00643 (2020). arXiv:2012.00643 https:\/\/arxiv.org\/abs\/2012.00643"},{"key":"e_1_3_3_21_2","volume-title":"Autonomous Vehicles Tests Without a Driver","year":"2022","unstructured":"DMV California 2022. Autonomous Vehicles Tests Without a Driver. Online Source. Department of Motor Vehicles, State of California. Retrieved from https:\/\/www.dmv.ca.gov\/portal\/vehicle-industry-services\/autonomous-vehicles\/testing-autonomous-vehicles-without-a-driver\/"},{"key":"e_1_3_3_22_2","first-page":"1","volume-title":"Conference on Robot Learning","author":"Dosovitskiy Alexey","year":"2017","unstructured":"Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. 2017. CARLA: An open urban driving simulator. In Conference on Robot Learning. PMLR, 1\u201316."},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.11648\/j.ajtas.20160501.11"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0184952"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338942"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/AITest55621.2022.00022"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2018.2881041"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00013"},{"key":"e_1_3_3_29_2","volume-title":"10th European Congress on Embedded Real Time Systems (ERTS)","author":"Gyllenhammar Magnus","year":"2020","unstructured":"Magnus Gyllenhammar, Rolf Johansson, Fredrik Warg, DeJiu Chen, Hans-Martin Heyn, Martin Sanfridson, Jan S\u00f6derberg, Anders Thorsen, and Stig Ursing. 2020. Towards an operational design domain that supports the safety argumentation of an automated driving system. In 10th European Congress on Embedded Real Time Systems (ERTS)."},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2022.3172442"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.apnr.2005.06.001"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.4271\/2018-01-1066"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/ITSC.2019.8917326"},{"key":"e_1_3_3_34_2","volume-title":"Road Vehicles\u2014Safety of the Intended Functionality","year":"2022","unstructured":"ISO 21448:2022(en) 2022. Road Vehicles\u2014Safety of the Intended Functionality. Standard. International Organization for Standardization. Retrieved from https:\/\/www.iso.org\/obp\/ui\/#iso:std:77490:en"},{"key":"e_1_3_3_35_2","volume-title":"Road Vehicles\u2014Functional Safety\u2014Part 1: Vocabulary","year":"2018","unstructured":"ISO 26262-1:2018(en) 2018. Road Vehicles\u2014Functional Safety\u2014Part 1: Vocabulary. Standard. International Organization for Standardization. Retrieved from https:\/\/www.iso.org\/obp\/ui\/#iso:std:iso:26262:-1:ed-2:v1:en"},{"key":"e_1_3_3_36_2","volume-title":"Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles","year":"2021","unstructured":"J3016_202104 2021. Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles. Standard. SAE International. Retrieved from https:\/\/www.sae.org\/standards\/content\/j3016_202104\/"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST49551.2021.00030"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/ITSC48978.2021.9564428"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.tra.2016.09.010"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIV.2018.2886678"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/RASSE54974.2022.9989690"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/QRS.2019.00058"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/ITSC48978.2021.9564591"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58920-2_11"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3550270"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE5003.2020.00012"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2019.106200"},{"key":"e_1_3_3_48_2","first-page":"64","volume-title":"Guidelines for Conducting Surveys in Software Engineering","author":"Lin\u00e5ker Johan","year":"2015","unstructured":"Johan Lin\u00e5ker, Sardar Sulaman, Martin H\u00f6st, and Rafael de Mello. 2015. Guidelines for Conducting Surveys in Software Engineering. Technical Report. Department of Computer Science, Lund University.64 pages. Retrieved from https:\/\/lup.lub.lu.se\/search\/files\/6062997\/5463412.pdf"},{"key":"e_1_3_3_49_2","unstructured":"Waymo LLC. 2023. Open Dataset\u2014Waymo. Retrieved from https:\/\/waymo.com\/open\/"},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","unstructured":"Guannan Lou Yao Deng Xi Zheng Mengshi Zhang and Tianyi Zhang. 2022. Testing of autonomous driving systems: Where are we and where should we go?(ESEC\/FSE 2022). Association for Computing Machinery New York NY 31\u201343. DOI:10.1145\/3540250.3549111","DOI":"10.1145\/3540250.3549111"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR59073.2023.00020"},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-88106-1_4"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678883"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.17763\/haer.62.3.8323320856251826"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1177\/1525822X02239573"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/IVS.2018.8500406"},{"key":"e_1_3_3_57_2","volume-title":"Moose Test - Wikipedia","year":"2023","unstructured":"Moose Test 2023. Moose Test - Wikipedia. Online Source. Wikipedia. Retrieved from https:\/\/en.wikipedia.org\/wiki\/Moose_test"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2022.3162601"},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/IV47402.2020.9304823"},{"key":"e_1_3_3_60_2","doi-asserted-by":"publisher","unstructured":"Vuong Nguyen Alessio Gambi Jasim Ahmed and Gordon Fraser. 2022. CRISCE: Towards generating test cases from accident sketches(ICSE\u201922). Association for Computing Machinery New York NY 339\u2013340. DOI:10.1145\/3510454.3528642","DOI":"10.1145\/3510454.3528642"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/AITEST52744.2021.00033"},{"key":"e_1_3_3_62_2","volume-title":"Laying Down Harmonised Rules On Artificial Intelligence (Artificial Intelligence Act) And Amending Certain Union Legislative Acts","year":"2021","unstructured":"Official Journal of the European Union 2021. Laying Down Harmonised Rules On Artificial Intelligence (Artificial Intelligence Act) And Amending Certain Union Legislative Acts. Regulation. European Parliament and the Council. Retrieved from https:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/?uri=celex%3A52021PC0206"},{"key":"e_1_3_3_63_2","volume-title":"Commission Implementing Regulation (EU) 2022\/1426","year":"2022","unstructured":"Official Journal of the European Union 2022. Commission Implementing Regulation (EU) 2022\/1426. Regulation. European Parliament and the Council. Retrieved from https:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/?uri=CELEX%3A32022R1426"},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jsr.2022.08.011"},{"key":"e_1_3_3_65_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.trpro.2020.03.003"},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1109\/EVER48776.2020.9242539"},{"key":"e_1_3_3_67_2","unstructured":"Felix Reisgys Johannes Plaum Andreas Schwarzhaupt and Eric Sax. 2022. Scenario-based X-in-the-loop test for development of driving automation(14. Uni-DAS e.V. Workshop Fahrerassistenz und automatisiertes Fahren)."},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.2993730"},{"key":"e_1_3_3_69_2","doi-asserted-by":"publisher","DOI":"10.3390\/s19030648"},{"key":"e_1_3_3_70_2","doi-asserted-by":"publisher","DOI":"10.1108\/01409171211210154"},{"key":"e_1_3_3_71_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-008-9102-8"},{"key":"e_1_3_3_72_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2021.3072739"},{"key":"e_1_3_3_73_2","volume-title":"Scenario-based Verification and Validation of Self-Driving Vehicles: Relevant Safety Metrics","year":"2022","unstructured":"Simens 2022. Scenario-based Verification and Validation of Self-Driving Vehicles: Relevant Safety Metrics. White Paper. Siemens Digital Industries Software."},{"key":"e_1_3_3_74_2","volume-title":"Critical Reasons for Crashes Investigated in the National Motor Vehicle Crash Causation Survey","author":"Singh Santokh","year":"2015","unstructured":"Santokh Singh. 2015. Critical Reasons for Crashes Investigated in the National Motor Vehicle Crash Causation Survey. Technical Report."},{"key":"e_1_3_3_75_2","doi-asserted-by":"publisher","DOI":"10.1109\/WAIN52551.2021.00018"},{"key":"e_1_3_3_76_2","doi-asserted-by":"publisher","unstructured":"Qunying Song Emelie Engstr\u00f6m and Per Runeson. 2023. Supplementary Materials. DOI:10.5281\/zenodo.10436801","DOI":"10.5281\/zenodo.10436801"},{"key":"e_1_3_3_77_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2022.107135"},{"key":"e_1_3_3_78_2","doi-asserted-by":"publisher","DOI":"10.1109\/AITEST52744.2021.00024"},{"key":"e_1_3_3_79_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11219-022-09604-2"},{"key":"e_1_3_3_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556968"},{"key":"e_1_3_3_81_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3202311"},{"key":"e_1_3_3_82_2","doi-asserted-by":"publisher","DOI":"10.1002\/smr.2386"},{"key":"e_1_3_3_83_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-020-09858-z"},{"key":"e_1_3_3_84_2","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2021.3136353"},{"key":"e_1_3_3_85_2","doi-asserted-by":"publisher","DOI":"10.1145\/3579642"},{"key":"e_1_3_3_86_2","doi-asserted-by":"publisher","DOI":"10.1109\/IVWorkshops54471.2021.9669219"},{"key":"e_1_3_3_87_2","doi-asserted-by":"publisher","DOI":"10.1109\/ITSC.2015.164"},{"key":"e_1_3_3_88_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11831-022-09788-7"},{"key":"e_1_3_3_89_2","doi-asserted-by":"publisher","DOI":"10.4271\/2020-01-1206"},{"key":"e_1_3_3_90_2","doi-asserted-by":"publisher","DOI":"10.1109\/IV47402.2020.9304609"},{"key":"e_1_3_3_91_2","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2022.3159500"},{"key":"e_1_3_3_92_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3170122"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3640334","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3640334","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:50:26Z","timestamp":1750287026000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3640334"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,18]]},"references-count":91,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,5,31]]}},"alternative-id":["10.1145\/3640334"],"URL":"https:\/\/doi.org\/10.1145\/3640334","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,18]]},"assertion":[{"value":"2023-04-11","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-12-21","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-04-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}