{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,18]],"date-time":"2026-05-18T23:23:01Z","timestamp":1779146581113,"version":"3.51.4"},"reference-count":77,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2023,4,26]],"date-time":"2023-04-26T00:00:00Z","timestamp":1682467200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001602","name":"Science Foundation Ireland","doi-asserted-by":"crossref","award":["20\/FFP-P\/8818 and 13\/RC\/2094_P2"],"award-info":[{"award-number":["20\/FFP-P\/8818 and 13\/RC\/2094_P2"]}],"id":[{"id":"10.13039\/501100001602","id-type":"DOI","asserted-by":"crossref"}]},{"name":"European Regional Development Fund through the Southern & Eastern Regional Operational Programme to Lero - the Science Foundation Ireland Research Centre for Software"},{"name":"ERATO HASUO Metamathematics for Systems Design","award":["JPMJER1603 JPMJER1603"],"award-info":[{"award-number":["JPMJER1603 JPMJER1603"]}]},{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"crossref","award":["20K23334"],"award-info":[{"award-number":["20K23334"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Engineerable AI Techniques for Practical Applications of High-Quality Machine Learning-based Systems","award":["JPMJMI20B8"],"award-info":[{"award-number":["JPMJMI20B8"]}]},{"name":"Research Activity Start-up","award":["20K23334 20K23334"],"award-info":[{"award-number":["20K23334 20K23334"]}]},{"name":"Paolo Arcaini and Fuyuki Ishikawa"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2023,7,31]]},"abstract":"<jats:p>Autonomous Driving Systems (ADSs) are promising, but must show they are secure and trustworthy before adoption. Simulation-based testing is a widely adopted approach, where the ADS is run in a simulated environment over specific scenarios. Coverage criteria specify what needs to be covered to consider the ADS sufficiently tested. However, existing criteria do not guarantee to exercise the different decisions that the ADS can make, which is essential to assess its correctness. ADSs usually compute their decisions using parameterised rule-based systems and cost functions, such as cost components or decision thresholds. In this article, we argue that the parameters characterise the decision process, as their values affect the ADS\u2019s final decisions. Therefore, we propose parameter coverage, a criterion requiring to cover the ADS\u2019s parameters. A scenario covers a parameter if changing its value leads to different simulation results, meaning it is relevant for the driving decisions made in the scenario. Since ADS simulators are slightly uncertain, we employ statistical methods to assess multiple simulation runs for execution difference and coverage. Experiments using the Autonomoose ADS show that the criterion discriminates between different scenarios and that the cost of computing coverage can be managed with suitable heuristics.<\/jats:p>","DOI":"10.1145\/3550270","type":"journal-article","created":{"date-parts":[[2022,7,22]],"date-time":"2022-07-22T11:18:57Z","timestamp":1658488737000},"page":"1-31","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":28,"title":["Parameter Coverage for Testing of Autonomous Driving Systems under Uncertainty"],"prefix":"10.1145","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0953-774X","authenticated-orcid":false,"given":"Thomas","family":"Laurent","sequence":"first","affiliation":[{"name":"Lero &amp; University College Dublin, Belfield, Dublin, Ireland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4212-7029","authenticated-orcid":false,"given":"Stefan","family":"Klikovits","sequence":"additional","affiliation":[{"name":"National Instituteof Informatics, Chiyoda-ku, Tokyo, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6253-4062","authenticated-orcid":false,"given":"Paolo","family":"Arcaini","sequence":"additional","affiliation":[{"name":"National Instituteof Informatics, Chiyoda-ku, Tokyo, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7725-2618","authenticated-orcid":false,"given":"Fuyuki","family":"Ishikawa","sequence":"additional","affiliation":[{"name":"National Instituteof Informatics, Chiyoda-ku, Tokyo, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2064-1238","authenticated-orcid":false,"given":"Anthony","family":"Ventresque","sequence":"additional","affiliation":[{"name":"Lero &amp; University College Dublin, Belfield, Dublin, Ireland"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,4,26]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Open Source Robotics Foundation. 2022. Robot Operating System (ROS). Retrieved from https:\/\/www.ros.org\/."},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST46399.2020.00020"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1016\/bs.adcom.2017.06.001"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/225058.225161"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1017\/9781316771273"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.4271\/2020-01-1204"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST49551.2021.00042"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST53961.2022.00037"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/1985793.1985795"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1486"},{"key":"e_1_3_2_12_2","unstructured":"Baidu. 2020. Apollo. Retrieved from https:\/\/github.com\/ApolloAuto\/apollo."},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-75632-5"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.sbspro.2012.06.1096"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/2970276.2970311"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180160"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238192"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST46399.2020.00045"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377930.3389827"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST49551.2021.00019"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1049\/sej.1994.0025"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.13140\/RG.2.2.18037.88803"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.13140\/RG.2.2.15521.30568"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.13140\/RG.2.2.11327.00165"},{"key":"e_1_3_2_25_2","unstructured":"Alex Davies. 2016. Google\u2019s Self-driving Car Caused Its First Crash. Retrieved from https:\/\/www.wired.com\/2016\/02\/googles-self-driving-car-may-caused-first-crash\/."},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.3685309"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-00262-6_4"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3314221.3314633"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338942"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330566"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00013"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2015.7102580"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/IV47402.2020.9304636"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-26601-1_5"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/ITSC.2019.8917326"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/ITSC.2015.327"},{"key":"e_1_3_2_37_2","unstructured":"Marko Ilievski. 2020. WiseBench: A Motion Planning Benchmarking Framework for Autonomous Vehicles. Retrieved from uwspace.uwaterloo.ca\/bitstream\/handle\/10012\/16422\/Ilievski_Marko.pdf."},{"key":"e_1_3_2_38_2","unstructured":"Waymo Inc. 2018. On the Road to Fully Self-Driving: Waymo Safety Report. Retrieved from storage.googleapis.com\/waymo-uploads\/files\/documents\/safety\/2021-12-waymo-safety-report.pdf."},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST49551.2021.00030"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST46399.2020.00018"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.7249\/j.ctt1btc0xw"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-016-9437-5"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/PRDC53464.2021.00011"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/APSEC48747.2019.00022"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICECCS51672.2020.00018"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678593"},{"key":"e_1_3_2_47_2","unstructured":"Thomas Laurent Stefan Klikovits Paolo Arcaini Fuyuki Ishikawa and Anthony Ventresque. 2022. Results repository. Retrieved from https:\/\/figshare.com\/s\/4a61edde5d37f01c9d68."},{"key":"e_1_3_2_48_2","unstructured":"Sam Levin and Julia Wong Carrie. 2018. Self-driving Uber kills Arizona woman in first fatal crash involving pedestrian. Retrieved from https:\/\/www.theguardian.com\/technology\/2018\/mar\/19\/uber-self-driving-car-kills-woman-arizona-tempe."},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2020.2970924"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE5003.2020.00012"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2019.106200"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678883"},{"key":"e_1_3_2_53_2","unstructured":"James Manyika Michael Chui Jacques Bughin Richard Dobbs Peter Bisson and Alex Marrs. 2013. Disruptive technologies: Advances that will transform life business and the global economy. Retrieved from https:\/\/www.mckinsey.com\/business-functions\/mckinsey-digital\/our-insights\/disruptive-technologies."},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW.2011.100"},{"key":"e_1_3_2_55_2","volume-title":"Parallel Algorithms for Real-time Motion Planning","author":"McNaughton Matthew","year":"2011","unstructured":"Matthew McNaughton. 2011. Parallel Algorithms for Real-time Motion Planning. Carnegie Mellon University."},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-10-1415-4_4"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/1883612.1883618"},{"key":"e_1_3_2_58_2","unstructured":"University of Waterloo. 2016. Autonomoose. Retrieved from https:\/\/www.autonomoose.net\/."},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2015.7102604"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1016\/bs.adcom.2018.03.015"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3476105"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2012.111"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-020-09881-0"},{"key":"e_1_3_2_64_2","article-title":"Definitions for terms related to on-road motor vehicle automated driving systems","year":"2014","unstructured":"SAE. 2014. Definitions for terms related to on-road motor vehicle automated driving systems. J3016, SAE Int. Stand. (2014).","journal-title":"J3016, SAE Int. Stand."},{"key":"e_1_3_2_65_2","volume-title":"Field and Service Robotics","author":"Shah Shital","year":"2017","unstructured":"Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. 2017. AirSim: High-fidelity visual and physical simulation for autonomous vehicles. In Field and Service Robotics. Retrieved from https:\/\/arxiv.org\/abs\/1705.05065."},{"key":"e_1_3_2_66_2","unstructured":"Jack Stewart. 2018. Tesla\u2019s Self-driving Autopilot Involved in Another Deadly Crash. Retrieved from https:\/\/www.wired.com\/story\/tesla-autopilot-self-driving-crash-california\/."},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW.2019.00058"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-78917-8_1"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1016\/bs.adcom.2015.11.004"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-45956-7_5"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.3390\/s21165397"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11219-012-9174-y"},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-662-48847-8_21"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-29044-2"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-99927-2_14"},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.2983149"},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2962027"},{"key":"e_1_3_2_78_2","unstructured":"Ding Zhao and Huei Peng. 2017. From the lab to the street: Solving the challenge of accelerating automated vehicle testing. Retrieved from http:\/\/arxiv.org\/abs\/1707.04792."}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3550270","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3550270","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T18:43:23Z","timestamp":1750272203000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3550270"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,26]]},"references-count":77,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,7,31]]}},"alternative-id":["10.1145\/3550270"],"URL":"https:\/\/doi.org\/10.1145\/3550270","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,4,26]]},"assertion":[{"value":"2021-12-23","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-07-10","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-04-26","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}