{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,13]],"date-time":"2026-03-13T04:09:32Z","timestamp":1773374972442,"version":"3.50.1"},"reference-count":52,"publisher":"Association for Computing Machinery (ACM)","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"abstract":"<jats:p>Testing Autonomous Driving Systems (ADS) is critical for validating their safety in operational environments. High-fidelity simulators enable the testing of ADS through virtual driving scenarios, especially those that are hazardous to replicate in real-world settings. However, existing testing approaches suffer from inadequate coverage of real-world traffic situations due to over-simplified modeling of vehicle movements (e.g., insufficient diversity in driving styles), resulting in undetected critical ADS failures. In this paper, we propose a testing framework to discover diverse failures of ADS in driving scenarios that embody real-world traffic complexity. The framework leverages advanced traffic simulation methods to encode vehicle movements and generates realistic yet safety-critical driving scenarios for ADS by mutating vehicle movements. To efficiently explore driving scenarios that pose different challenges for ADS and expose diverse ADS failures, this framework further leverages a dynamic prioritization mechanism that prioritizes vehicle movements likely to trigger unique ADS behaviors. Specifically, we propose a method to estimate the possibility based on encoded vehicle movements. We implement this framework and evaluate it with three representative ADS from the famous CARLA leaderboard. Empirical evaluation demonstrates that the proposed approach discovers more unique failures of ADS than existing testing frameworks.<\/jats:p>","DOI":"10.1145\/3727875","type":"journal-article","created":{"date-parts":[[2025,4,9]],"date-time":"2025-04-09T17:20:41Z","timestamp":1744219241000},"update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Autonomous Driving System Testing via Diversity-Oriented Driving Scenario Exploration"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1109-9766","authenticated-orcid":false,"given":"Xinyu","family":"Ji","sequence":"first","affiliation":[{"name":"Sun Yat-sen University and The Hong Kong Polytechnic University, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5321-5740","authenticated-orcid":false,"given":"Lei","family":"Xue","sequence":"additional","affiliation":[{"name":"Sun Yat-sen University and Guangdong Provincial Key Laboratory of Information Security Technology, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1735-2331","authenticated-orcid":false,"given":"Zhijian","family":"He","sequence":"additional","affiliation":[{"name":"Shenzhen Technology University, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9082-3208","authenticated-orcid":false,"given":"Xiapu","family":"Luo","sequence":"additional","affiliation":[{"name":"The Hong Kong Polytechnic University, China"}]}],"member":"320","published-online":{"date-parts":[[2025,4,9]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"2019 International Conference on Robotics and Automation (ICRA). IEEE, 8271\u20138277","author":"Abeysirigoonawardena Yasasa","year":"2019","unstructured":"Yasasa Abeysirigoonawardena, Florian Shkurti, and Gregory Dudek. 2019. Generating adversarial driving scenarios in high-fidelity simulators. In 2019 International Conference on Robotics and Automation (ICRA). IEEE, 8271\u20138277."},{"key":"e_1_2_1_2_1","volume-title":"Safety-Critical and Motion Sickness-Inducing Scenarios for Autonomous Vehicles. arXiv preprint arXiv:2112.09725","author":"Almanee Sumaya","year":"2021","unstructured":"Sumaya Almanee, Xiafa Wu, Yuqi Huai, Qi\u00a0Alfred Chen, and Joshua Garcia. 2021. scenoRITA: Generating Less-Redundant, Safety-Critical and Motion Sickness-Inducing Scenarios for Autonomous Vehicles. arXiv preprint arXiv:2112.09725 (2021)."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1985793.1985795"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2970276.2970311"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238192"},{"key":"e_1_2_1_6_1","volume-title":"International journal of transportation science and technology 6, 3","author":"Chan Ching-Yao","year":"2017","unstructured":"Ching-Yao Chan. 2017. Advancements, prospects, and impacts of automated driving systems. International journal of transportation science and technology 6, 3 (2017), 208\u2013216."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00895"},{"key":"e_1_2_1_8_1","volume-title":"Adversarial evaluation of autonomous vehicles in lane-change scenarios","author":"Chen Baiming","year":"2021","unstructured":"Baiming Chen, Xiang Chen, Qiong Wu, and Liang Li. 2021a. Adversarial evaluation of autonomous vehicles in lane-change scenarios. IEEE Transactions on Intelligent Transportation Systems (2021)."},{"key":"e_1_2_1_9_1","doi-asserted-by":"crossref","unstructured":"Dian Chen Vladlen Koltun and Philipp Kr\u00e4henb\u00fchl. 2021b. Learning to drive from a world on rails. In ICCV.","DOI":"10.1109\/ICCV48922.2021.01530"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3597926.3598072"},{"key":"e_1_2_1_11_1","unstructured":"Papers\u00a0With Code. 2024. Autonomous Driving on CARLA Leaderboard. https:\/\/paperswithcode.com\/sota\/autonomous-driving-on-carla-leaderboard."},{"key":"e_1_2_1_12_1","unstructured":"Cruise LLC. 2024. Cruise Releases Third-Party Findings Regarding October 2. https:\/\/www.getcruise.com\/news\/blog\/2024\/cruise-releases-third-party-findings-regarding-october-2\/."},{"key":"e_1_2_1_13_1","volume-title":"Scenario-Based Test Reduction and Prioritization for Multi-Module Autonomous Driving Systems. arXiv preprint arXiv:2209.01546","author":"Deng Yao","year":"2022","unstructured":"Yao Deng, Xi Zheng, Mengshi Zhang, Guannan Lou, and Tianyi Zhang. 2022b. Scenario-Based Test Reduction and Prioritization for Multi-Module Autonomous Driving Systems. arXiv preprint arXiv:2209.01546 (2022)."},{"key":"e_1_2_1_14_1","doi-asserted-by":"crossref","first-page":"1964","DOI":"10.1109\/TSE.2022.3206427","article-title":"A declarative metamorphic testing framework for autonomous driving","volume":"49","author":"Deng Yao","year":"2022","unstructured":"Yao Deng, Xi Zheng, Tianyi Zhang, Huai Liu, Guannan Lou, Miryung Kim, and Tsong\u00a0Yueh Chen. 2022a. A declarative metamorphic testing framework for autonomous driving. IEEE Transactions on Software Engineering 49, 4 (2022), 1964\u20131982.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2021.3058873"},{"key":"e_1_2_1_16_1","unstructured":"DoFuzz. 2025. Online repository. https:\/\/github.com\/ADSTestKit\/DoFuzz."},{"key":"e_1_2_1_17_1","volume-title":"Proceedings of the 1st Annual Conference on Robot Learning. 1\u201316","author":"Dosovitskiy Alexey","year":"2017","unstructured":"Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. 2017. CARLA: An Open Urban Driving Simulator. In Proceedings of the 1st Annual Conference on Robot Learning. 1\u201316."},{"key":"e_1_2_1_18_1","volume-title":"et\u00a0al","author":"Gulino Cole","year":"2024","unstructured":"Cole Gulino, Justin Fu, Wenjie Luo, George Tucker, Eli Bronstein, Yiren Lu, Jean Harb, Xinlei Pan, Yan Wang, Xiangyu Chen, et\u00a0al. 2024. Waymax: An accelerated, data-driven simulator for large-scale autonomous driving research. Advances in Neural Information Processing Systems 36 (2024)."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00029"},{"key":"e_1_2_1_20_1","volume-title":"Coverage-based scene fuzzing for virtual autonomous driving testing. arXiv preprint arXiv:2106.00873","author":"Hu Zhisheng","year":"2021","unstructured":"Zhisheng Hu, Shengjian Guo, Zhenyu Zhong, and Kang Li. 2021. Coverage-based scene fuzzing for virtual autonomous driving testing. arXiv preprint arXiv:2106.00873 (2021)."},{"key":"e_1_2_1_21_1","unstructured":"Jack Fitzgerald. 2023. Tesla Recalls 362 758 Cars over Full Self-Driving Crash Risk. https:\/\/www.caranddriver.com\/news\/a42941490\/tesla-full-self-driving-crash-risk-recall\/."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00108"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3417065"},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security. 1753\u20131767","author":"Kim Seulbae","unstructured":"Seulbae Kim, Major Liu, Junghwan\u00a0\u201cJohn\u201d Rhee, Yuseok Jeon, Yonghwi Kwon, and Chung\u00a0Hwan Kim. 2022. Drivefuzz: Discovering autonomous driving bugs through driving quality-guided fuzzing. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security. 1753\u20131767."},{"key":"e_1_2_1_25_1","volume-title":"2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE). IEEE, 25\u201336","author":"Li Guanpeng","year":"2020","unstructured":"Guanpeng Li, Yiran Li, Saurabh Jha, Timothy Tsai, Michael Sullivan, Siva Kumar\u00a0Sastry Hari, Zbigniew Kalbarczyk, and Ravishankar Iyer. 2020. AV-FUZZER: Finding safety violations in autonomous driving systems. In 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE). IEEE, 25\u201336."},{"key":"e_1_2_1_26_1","volume-title":"International Conference on Machine Learning. PMLR, 6471\u20136482","author":"Li Zhong","year":"2021","unstructured":"Zhong Li, Minxue Pan, Tian Zhang, and Xuandong Li. 2021. Testing dnn-based autonomous driving systems under critical environmental conditions. In International Conference on Machine Learning. PMLR, 6471\u20136482."},{"key":"e_1_2_1_27_1","volume-title":"European Conference on Computer Vision. Springer, 541\u2013556","author":"Liang Ming","year":"2020","unstructured":"Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. 2020. Learning lane graph representations for motion forecasting. In European Conference on Computer Vision. Springer, 541\u2013556."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678883"},{"key":"e_1_2_1_29_1","unstructured":"Manu Granda. 2023. Safety concerns hinder the advancement of driverless cars in the US. https:\/\/english.elpais.com\/economy-and-business\/2023-12-27\/safety-concerns-hinder-the-advancement-of-driverless-cars-in-the-us.html."},{"key":"e_1_2_1_30_1","unstructured":"Mauricio Pe\u00f1a. 2024. Voluntary recall of our previous software. https:\/\/waymo.com\/blog\/2024\/02\/voluntary-recall-of-our-previous-software\/."},{"key":"e_1_2_1_31_1","unstructured":"Muskan Ali. 2023. NHTSA Crash Report Highlights Safety Concerns With Self-Driving Cars. https:\/\/www.sommerspc.com\/blog\/2023\/08\/nhtsa-crash-report-highlights-safety-concerns-with-self-driving-cars\/."},{"key":"e_1_2_1_32_1","volume-title":"The kinematic bicycle model: A consistent model for planning feasible trajectories for autonomous vehicles?. In 2017 IEEE intelligent vehicles symposium (IV)","author":"Polack Philip","unstructured":"Philip Polack, Florent Altch\u00e9, Brigitte d\u2019Andr\u00e9a Novel, and Arnaud de La\u00a0Fortelle. 2017. The kinematic bicycle model: A consistent model for planning feasible trajectories for autonomous vehicles?. In 2017 IEEE intelligent vehicles symposium (IV). IEEE, 812\u2013818."},{"key":"e_1_2_1_33_1","volume-title":"Multi-Modal Fusion Transformer for End-to-End Autonomous Driving. In Conference on Computer Vision and Pattern Recognition (CVPR).","author":"Prakash Aditya","year":"2021","unstructured":"Aditya Prakash, Kashyap Chitta, and Andreas Geiger. 2021. Multi-Modal Fusion Transformer for End-to-End Autonomous Driving. In Conference on Computer Vision and Pattern Recognition (CVPR)."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33014780"},{"key":"e_1_2_1_35_1","volume-title":"Nonlinear systems: analysis, stability, and control. Vol.\u00a010","author":"Sastry Shankar","unstructured":"Shankar Sastry. 2013. Nonlinear systems: analysis, stability, and control. Vol.\u00a010. Springer Science & Business Media."},{"key":"e_1_2_1_36_1","volume-title":"Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer. arXiv preprint arXiv:2207.14024","author":"Shao Hao","year":"2022","unstructured":"Hao Shao, Letian Wang, RuoBing Chen, Hongsheng Li, and Yu Liu. 2022. Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer. arXiv preprint arXiv:2207.14024 (2022)."},{"key":"e_1_2_1_37_1","volume-title":"Calculus: Early Transcendentals (th ed.). Cengage Learning.","author":"Stewart James","year":"2015","unstructured":"James Stewart. 2015. Calculus: Early Transcendentals (th ed.). Cengage Learning."},{"key":"e_1_2_1_38_1","volume-title":"2022 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 11416\u201311423","author":"Sun Qiao","year":"2022","unstructured":"Qiao Sun, Xin Huang, Brian\u00a0C Williams, and Hang Zhao. 2022. InterSim: Interactive traffic simulation via explicit relation modeling. In 2022 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 11416\u201311423."},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 10400\u201310409","author":"Suo Simon","year":"2021","unstructured":"Simon Suo, Sebastian Regalado, Sergio Casas, and Raquel Urtasun. 2021. Trafficsim: Learning to simulate realistic multi-agent behaviors. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 10400\u201310409."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678735"},{"key":"e_1_2_1_41_1","volume-title":"Intel Autonomous\u00a0Agents Lab","author":"Team CARLA","year":"2023","unstructured":"CARLA Team, Intel Autonomous\u00a0Agents Lab, Embodied\u00a0AI Foundation, and AlphaDrive. 2023. CARLA Autonomous Driving Leaderboard. https:\/\/leaderboard.carla.org\/."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549100"},{"key":"e_1_2_1_43_1","volume-title":"37th IEEE\/ACM International Conference on Automated Software Engineering. 1\u201312","author":"Tian Haoxiang","year":"2022","unstructured":"Haoxiang Tian, Guoquan Wu, Jiren Yan, Yan Jiang, Jun Wei, Wei Chen, Shuo Li, and Dan Ye. 2022b. Generating Critical Test Scenarios for Autonomous Driving Systems via Influential Behavior Patterns. In 37th IEEE\/ACM International Conference on Automated Software Engineering. 1\u201312."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00978"},{"key":"e_1_2_1_46_1","volume-title":"2017 IEEE International symposium on software reliability engineering workshops (ISSREW). IEEE, 294\u2013301","author":"Wolschke Christian","year":"2017","unstructured":"Christian Wolschke, Thomas Kuhn, Dieter Rombach, and Peter Liggesmeyer. 2017. Observation based creation of minimal test suites for autonomous vehicles. In 2017 IEEE International symposium on software reliability engineering workshops (ISSREW). IEEE, 294\u2013301."},{"key":"e_1_2_1_47_1","volume-title":"Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline. arXiv preprint arXiv:2206.08129","author":"Wu Penghao","year":"2022","unstructured":"Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, and Yu Qiao. 2022. Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline. arXiv preprint arXiv:2206.08129 (2022)."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238187"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01473"},{"key":"e_1_2_1_50_1","volume-title":"Neural network guided evolutionary fuzzing for finding traffic violations of autonomous vehicles","author":"Zhong Ziyuan","year":"2022","unstructured":"Ziyuan Zhong, Gail Kaiser, and Baishakhi Ray. 2022. Neural network guided evolutionary fuzzing for finding traffic violations of autonomous vehicles. IEEE Transactions on Software Engineering (2022)."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380422"},{"key":"e_1_2_1_52_1","volume-title":"International Conference on Learning Representations.","author":"Zhu Deyao","year":"2020","unstructured":"Deyao Zhu, Mohamed Zahran, Li\u00a0Erran Li, and Mohamed Elhoseiny. 2020. Halentnet: Multimodal trajectory forecasting with hallucinative intents. In International Conference on Learning Representations."}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3727875","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,4,9]],"date-time":"2025-04-09T17:20:53Z","timestamp":1744219253000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3727875"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4,9]]},"references-count":52,"alternative-id":["10.1145\/3727875"],"URL":"https:\/\/doi.org\/10.1145\/3727875","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,4,9]]},"assertion":[{"value":"2024-03-31","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-01-09","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-04-09","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"3727875"}}