{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,29]],"date-time":"2026-07-29T11:23:36Z","timestamp":1785324216260,"version":"3.55.0"},"reference-count":88,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2023,4,26]],"date-time":"2023-04-26T00:00:00Z","timestamp":1682467200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,4,26]],"date-time":"2023-04-26T00:00:00Z","timestamp":1682467200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100007601","name":"horizon 2020","doi-asserted-by":"publisher","award":["957254-COSMOS"],"award-info":[{"award-number":["957254-COSMOS"]}],"id":[{"id":"10.13039\/501100007601","id-type":"DOI","asserted-by":"publisher"}]},{"name":"ZHAW Zurich University of Applied Sciences"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2023,5]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Simulation platforms facilitate the development of emerging Cyber-Physical Systems (CPS) like self-driving cars (SDC) because they are more efficient and less dangerous than field operational test cases. Despite this, thoroughly testing SDCs in simulated environments remains challenging because SDCs must be tested in a sheer amount of long-running test cases. Past results on software testing optimization have shown that not all the test cases contribute equally to establishing confidence in test subjects\u2019 quality and reliability, and the execution of \u201csafe and uninformative\u201d test cases can be skipped to reduce testing effort. However, this problem is only partially addressed in the context of SDC simulation platforms. In this paper, we investigate test selection strategies to increase the cost-effectiveness of simulation-based testing in the context of SDCs. We propose an approach called <jats:bold>SDC-Scissor<\/jats:bold> (<jats:bold>SDC<\/jats:bold> co<jats:bold>S<\/jats:bold> t-effe<jats:bold>C<\/jats:bold> t<jats:bold>I<\/jats:bold> ve te<jats:bold>S<\/jats:bold> t <jats:bold>S<\/jats:bold> elect<jats:bold>OR<\/jats:bold>) that leverages Machine Learning (ML) strategies to identify and skip test cases that are unlikely to detect faults in SDCs before executing them. Our evaluation shows that SDC-Scissor outperforms the baselines. With the Logistic model, we achieve an accuracy of 70%, a precision of 65%, and a recall of 80% in selecting tests leading to a fault and improved testing cost-effectiveness. Specifically, SDC-Scissor avoided the execution of 50% of <jats:italic>unnecessary<\/jats:italic> tests as well as outperformed two baseline strategies. Complementary to existing work, we also integrated SDC-Scissor into the context of an industrial organization in the automotive domain to demonstrate how it can be used in industrial settings.<\/jats:p>","DOI":"10.1007\/s10664-023-10286-y","type":"journal-article","created":{"date-parts":[[2023,4,26]],"date-time":"2023-04-26T09:03:20Z","timestamp":1682499800000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":64,"title":["Machine learning-based test selection for simulation-based testing of self-driving cars software"],"prefix":"10.1007","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3987-0276","authenticated-orcid":false,"given":"Christian","family":"Birchler","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0354-9747","authenticated-orcid":false,"given":"Sajad","family":"Khatiri","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bill","family":"Bosshard","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0132-6497","authenticated-orcid":false,"given":"Alessio","family":"Gambi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4120-626X","authenticated-orcid":false,"given":"Sebastiano","family":"Panichella","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,4,26]]},"reference":[{"key":"10286_CR1","doi-asserted-by":"publisher","unstructured":"Abdessalem RB, Nejati S, Briand LC, Stifter T (2016) Testing advanced driver assistance systems using multi-objective search and neural networks. In: Lo D, Apel S, Khurshid S (eds) Proceedings of the 31st IEEE\/ACM international conference on automated software engineering, ASE 2016, September 3-7. ACM, Singapore, pp 63\u201374. https:\/\/doi.org\/10.1145\/2970276.2970311","DOI":"10.1145\/2970276.2970311"},{"key":"10286_CR2","doi-asserted-by":"publisher","unstructured":"Abdessalem RB, Nejati S, Briand LC, Stifter T (2018a) Testing vision-based control systems using learnable evolutionary algorithms. In: Proceedings of the 40th international conference on software engineering. ACM, pp 1016\u20131026. https:\/\/doi.org\/10.1145\/3180155.3180160","DOI":"10.1145\/3180155.3180160"},{"key":"10286_CR3","doi-asserted-by":"publisher","unstructured":"Abdessalem RB, Panichella A, Nejati S, Briand LC, Stifter T (2018b) Testing autonomous cars for feature interaction failures using many-objective search. In: Huchard M, K\u00e4stner C, Fraser G (eds) Proceedings of the 33rd ACM\/IEEE international conference on automated software engineering, ASE 2018, September 3-7, 2018. IEEE, ACM, Montpellier, France, pp 143\u2013154. https:\/\/doi.org\/10.1145\/3238147.3238192","DOI":"10.1145\/3238147.3238192"},{"key":"10286_CR4","unstructured":"Academies of Sciences (2017) A 21st century cyber-physical systems education. National Academies Press"},{"key":"10286_CR5","doi-asserted-by":"publisher","first-page":"e803","DOI":"10.7717\/peerj-cs.803","volume":"8","author":"M Adnan","year":"2022","unstructured":"Adnan M, Alarood AA, Uddin MI, ur Rehman I (2022) Utilizing grid search cross-validation with adaptive boosting for augmenting performance of machine learning models. PeerJ Comput Sci 8:e803. https:\/\/doi.org\/10.7717\/peerj-cs.803","journal-title":"PeerJ Comput Sci"},{"key":"10286_CR6","doi-asserted-by":"crossref","unstructured":"Afzal A, Katz DS, Goues CL, Timperley CS (2020) A study on the challenges of using robotics simulators for testing","DOI":"10.1109\/ICST46399.2020.00020"},{"key":"10286_CR7","doi-asserted-by":"publisher","unstructured":"Althoff M, Koschi M, Manzinger S (2017) Commonroad: Composable benchmarks for motion planning on roads. In: IEEE intelligent vehicles symposium, IV 2017, June 11-14, 2017. IEEE, Los Angeles, CA, USA, pp 719\u2013726. https:\/\/doi.org\/10.1109\/IVS.2017.7995802","DOI":"10.1109\/IVS.2017.7995802"},{"issue":"3","key":"10286_CR8","doi-asserted-by":"publisher","first-page":"219","DOI":"10.1002\/stvr.1486","volume":"24","author":"A Arcuri","year":"2014","unstructured":"Arcuri A, Briand LC (2014) A hitchhiker\u2019s guide to statistical tests for assessing randomized algorithms in software engineering. Softw Test Verification Reliab 24(3):219\u2013250. https:\/\/doi.org\/10.1002\/stvr.1486","journal-title":"Softw Test Verification Reliab"},{"key":"10286_CR9","doi-asserted-by":"crossref","unstructured":"Arrieta A, Wang S, Arruabarrena A, Markiegi U, Sagardui G, Etxeberria L (2018a) Multi-objective black-box test case selection for cost-effectively testing simulation models. In: Proceedings of the genetic and evolutionary computation conference, pp 1411\u20131418","DOI":"10.1145\/3205455.3205490"},{"key":"10286_CR10","doi-asserted-by":"publisher","unstructured":"Arrieta A, Wang S, Markiegi U, Sagardui G, Etxeberria L (2018b) Employing multi-objective search to enhance reactive test case generation and prioritization for testing industrial cyber-physical systems. IEEE Trans Ind Inform 14(3):1055\u20131066. https:\/\/doi.org\/10.1109\/TII.2017.2788019","DOI":"10.1109\/TII.2017.2788019"},{"key":"10286_CR11","doi-asserted-by":"publisher","unstructured":"Arrieta A, Wang S, Sagardui G, Etxeberria L (2016) Search-based test case selection of cyber-physical system product lines for simulation-based validation. In: Mei H (ed) Proceedings of the 20th international systems and software product line conference, SPLC 2016, September 16-23, 2016. ACM, Beijing, China, pp 297\u2013306, DOI https:\/\/doi.org\/10.1145\/2934466.2946046","DOI":"10.1145\/2934466.2946046"},{"key":"10286_CR12","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.jss.2018.09.055","volume":"149","author":"A Arrieta","year":"2019","unstructured":"Arrieta A, Wang S, Sagardui G, Etxeberria L (2019) Search-based test case prioritization for simulation-based testing of cyber-physical system product lines. J Syst Softw 149:1\u201334. https:\/\/doi.org\/10.1016\/j.jss.2018.09.055","journal-title":"J Syst Softw"},{"key":"10286_CR13","volume-title":"Modern Information Retrieval - the concepts and technology behind search","author":"R Baeza-Yates","year":"2011","unstructured":"Baeza-Yates R, Ribeiro-Neto BA (2011) Modern Information Retrieval - the concepts and technology behind search, 2nd edn. Pearson Education Ltd., Harlow, England. http:\/\/www.mir2ed.org\/","edition":"2nd edn."},{"issue":"1","key":"10286_CR14","first-page":"161","volume":"12","author":"R Baheti","year":"2011","unstructured":"Baheti R, Gill H (2011) Cyber-physical systems. Impact Control Technol 12(1):161\u2013166","journal-title":"Impact Control Technol"},{"key":"10286_CR15","unstructured":"BeamNG GmbH (2022) BeamNG.tech. https:\/\/www.beamng.gmbh\/research. Accessed 11 Oct 2018"},{"key":"10286_CR16","unstructured":"Bergstra J, Bardenet R, Bengio Y, K\u00e9gl B (2011) Algorithms for hyper-parameter optimization. In: Proceedings of the 24th international conference on neural information processing systems, NIPS\u201911. Curran Associates Inc., Red Hook, NY, USA, pp 2546\u20132554"},{"key":"10286_CR17","doi-asserted-by":"publisher","first-page":"281","DOI":"10.5555\/2503308.2188395","volume":"13","author":"J Bergstra","year":"2012","unstructured":"Bergstra J, Bengio Y (2012) Random search for hyper-parameter optimization. J Mach Learn Res 13:281\u2013305. https:\/\/doi.org\/10.5555\/2503308.2188395","journal-title":"J Mach Learn Res"},{"key":"10286_CR18","doi-asserted-by":"publisher","unstructured":"Bettenburg N, Just S, Schr\u00f6ter A., Wei\u00df C, Premraj R, Zimmermann T (2007) Quality of bug reports in eclipse. In: Cheng L, Orso A, Robillard MP (eds) Proceedings of the 2007 OOPSLA workshop on Eclipse Technology eXchange, ETX 2007, October 21, 2007. ACM, Montreal, Quebec, Canada, pp 21\u201325, DOI https:\/\/doi.org\/10.1145\/1328279.1328284","DOI":"10.1145\/1328279.1328284"},{"key":"10286_CR19","doi-asserted-by":"crossref","unstructured":"Bezerra MER, Oliveira ALI, Meira SRL (2007) A constructive RBF neural network for estimating the probability of defects in software modules. In: 2007 international joint conference on neural networks, pp 2869\u20132874","DOI":"10.1109\/IJCNN.2007.4371415"},{"key":"10286_CR20","doi-asserted-by":"publisher","unstructured":"Birchler C, Ganz N, Khatiri S, Gambi A, Panichella S (2023) Cost-effective simulation-based test selection in self-driving cars software. Science of Computer Programming (SCP). https:\/\/doi.org\/10.1016\/j.scico.2023.102926","DOI":"10.1016\/j.scico.2023.102926"},{"key":"10286_CR21","doi-asserted-by":"publisher","unstructured":"Birchler C, Ganz N, Khatiri S, Gambi A, Panichella S (2022) Cost-effective simulation-based test selection in self-driving cars software with SDC-scissor. In: 2022 IEEE international conference on software analysis, evolution and reengineering (SANER), pp 164\u2013168. https:\/\/doi.org\/10.1109\/SANER53432.2022.00030","DOI":"10.1109\/SANER53432.2022.00030"},{"key":"10286_CR22","doi-asserted-by":"publisher","unstructured":"Birchler C, Khatiri S, Derakhshanfar P, Panichella S, Panichella A (2022c) Single and multi-objective test cases prioritization for self-driving cars in virtual environments. ACM Trans Softw Eng Methodol. https:\/\/doi.org\/10.1145\/3533818","DOI":"10.1145\/3533818"},{"key":"10286_CR23","doi-asserted-by":"publisher","unstructured":"Bondi E, Dey D, Kapoor A, Piavis J, Shah S, Fang F, Dilkina B, Hannaford R, Iyer A, Joppa L, Tambe M (2018) AirSim-w: A simulation environment for wildlife conservation with UAVs. In: Zegura EW (ed) Proceedings of the 1st ACM SIGCAS conference on computing and sustainable societies, COMPASS. ACM, pp 40:1\u201340:12. https:\/\/doi.org\/10.1145\/3209811.3209880","DOI":"10.1145\/3209811.3209880"},{"key":"10286_CR24","doi-asserted-by":"publisher","unstructured":"Boumiza S, Braham R (2019) An anomaly detector for can bus networks in autonomous cars based on neural networks. In: 2019 international conference on wireless and mobile computing, networking and communications (WiMob), pp 1\u20136. https:\/\/doi.org\/10.1109\/WiMOB.2019.8923315","DOI":"10.1109\/WiMOB.2019.8923315"},{"key":"10286_CR25","doi-asserted-by":"publisher","unstructured":"Briand L, Nejati S, Sabetzadeh M, Bianculli D (2016) Testing the untestable: Model testing of complex software-intensive systems. In: Dillon LK, Visser W, Williams LA (eds) Proc. int\u2019l conf on software engineering (ICSE \u2013 Companion). ACM, pp 789\u2013792. https:\/\/doi.org\/10.1145\/2889160.2889212","DOI":"10.1145\/2889160.2889212"},{"key":"10286_CR26","doi-asserted-by":"crossref","unstructured":"Canfora G, De Lucia A, Di Penta M, Oliveto R, Panichella A, Panichella S (2013) Multi-objective cross-project defect prediction. In: 2013 IEEE 6th international conference on software testing, verification and validation (ICST). IEEE, pp 252\u2013261","DOI":"10.1109\/ICST.2013.38"},{"key":"10286_CR27","doi-asserted-by":"publisher","unstructured":"Canfora G, Lucia AD, Penta MD, Oliveto R, Panichella A, Panichella S (2013) Multi-objective cross-project defect prediction. In: 6th IEEE international conference on software testing, verification and validation, ICST 2013, March 18-22, 2013. IEEE Computer Society, Luxembourg, pp 252\u2013261. https:\/\/doi.org\/10.1109\/ICST.2013.38","DOI":"10.1109\/ICST.2013.38"},{"issue":"4","key":"10286_CR28","doi-asserted-by":"publisher","first-page":"426","DOI":"10.1002\/stvr.1570","volume":"25","author":"G Canfora","year":"2015","unstructured":"Canfora G, Lucia AD, Penta MD, Oliveto R, Panichella A, Panichella S (2015) Defect prediction as a multiobjective optimization problem. Softw Test Verification Reliab 25(4):426\u2013459","journal-title":"Softw Test Verification Reliab"},{"key":"10286_CR29","doi-asserted-by":"crossref","unstructured":"Caruana R, Niculescu-mizil A (2006) An empirical comparison of supervised learning algorithms. In: In Proc. 23 rd Intl. Conf. Machine learning (ICML\u201906), pp 161\u2013168","DOI":"10.1145\/1143844.1143865"},{"key":"10286_CR30","doi-asserted-by":"publisher","unstructured":"Castellano E, Cetinkaya A, Thanh CH, Klikovits S, Zhang X, Arcaini P (2021) Frenetic at the SBST 2021 tool competition. In: 14th IEEE\/ACM international workshop on search-based software testing, SBST 2021, May 31, 2021. IEEE, Madrid, Spain, pp 36\u201337. https:\/\/doi.org\/10.1109\/SBST52555.2021.00016","DOI":"10.1109\/SBST52555.2021.00016"},{"key":"10286_CR31","doi-asserted-by":"crossref","unstructured":"Ceylan E, Kutlubay FO, Bener AB (2006) Software defect identification using machine learning techniques. In: 32nd EUROMICRO conference on software engineering and advanced applications (EUROMICRO\u201906), pp 240\u2013247","DOI":"10.1109\/EUROMICRO.2006.56"},{"issue":"03","key":"10286_CR32","doi-asserted-by":"publisher","first-page":"1750012","DOI":"10.1142\/S2424862217500129","volume":"02","author":"H Chen","year":"2017","unstructured":"Chen H (2017) Applications of cyber-physical system: A literature review. J Ind Integr Manag 02(03):1750012","journal-title":"J Ind Integr Manag"},{"issue":"3","key":"10286_CR33","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1016\/S0020-0190(96)00135-4","volume":"60","author":"TY Chen","year":"1996","unstructured":"Chen TY, Lau MF (1996) Dividing strategies for the optimization of a test suite. Inf Process Lett 60(3):135\u2013141","journal-title":"Inf Process Lett"},{"key":"10286_CR34","unstructured":"CIA (2017) History of can technology. https:\/\/www.can-cia.org\/can-knowledge\/can\/can-history\/. Accessed 8 Nov 2022"},{"key":"10286_CR35","unstructured":"CNX O (2021) Openstax university physics. http:\/\/cnx.org\/contents\/d50f6e32-0fda-46ef-a362-9bd36ca7c97d@10.16. Accessed 8 Nov 2022"},{"key":"10286_CR36","doi-asserted-by":"crossref","unstructured":"Dalboni M, Soldati A (2019) Soft-body modeling: A scalable and efficient formulation for control-oriented simulation of electric vehicles. In: IEEE transportation electrification conference and expo (ITEC), pp 1\u20136","DOI":"10.1109\/ITEC.2019.8790473"},{"key":"10286_CR37","doi-asserted-by":"crossref","unstructured":"Devroey X, Gambi A, Galeotti JP, Just R, Kifetew F, Panichella A, Panichella S (2022) Juge: An infrastructure for benchmarking java unit test generators. Software Testing Verification and Reliability","DOI":"10.1002\/stvr.1838"},{"key":"10286_CR38","unstructured":"Dosovitskiy A, Ros G, Codevilla F, L\u00f3pez AM, Koltun V (2017) CARLA: an open urban driving simulator. In: 1st annual conference on robot learning, CoRL 2017, Proceedings of Machine Learning Research, vol 78. PMLR, pp 1\u201316. http:\/\/proceedings.mlr.press\/v78\/dosovitskiy17a.html"},{"issue":"10","key":"10286_CR39","doi-asserted-by":"publisher","first-page":"1092","DOI":"10.1016\/j.infsof.2012.04.007","volume":"54","author":"F Elberzhager","year":"2012","unstructured":"Elberzhager F, Rosbach A, M\u00fcnch J, Eschbach R (2012) Reducing test effort: A systematic mapping study on existing approaches. Inf Softw Technol 54(10):1092\u20131106. https:\/\/doi.org\/10.1016\/j.infsof.2012.04.007","journal-title":"Inf Softw Technol"},{"key":"10286_CR40","first-page":"1305","volume-title":"Weka: A machine learning workbench for data mining","author":"E Frank","year":"2005","unstructured":"Frank E, Hall MA, Holmes G, Kirkby R, Pfahringer B, Witten IH (2005) Weka: A machine learning workbench for data mining. Springer, Berlin, pp 1305\u20131314. http:\/\/researchcommons.waikato.ac.nz\/handle\/10289\/1497"},{"key":"10286_CR41","doi-asserted-by":"publisher","unstructured":"Gambi A, Huynh T, Fraser G (2019) Generating effective test cases for self-driving cars from police reports. In: Dumas M, Pfahl D, Apel S, Russo A (eds) Proceedings of the 2019 27th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering - ESEC\/FSE 2019. ACM Press, pp 257\u2013267. https:\/\/doi.org\/10.1145\/3338906.3338942","DOI":"10.1145\/3338906.3338942"},{"key":"10286_CR42","doi-asserted-by":"publisher","unstructured":"Gambi A, Jahangirova G, Riccio V, Zampetti F (2022) SBST tool competition 2022. In: 15th IEEE\/ACM international workshop on search-based software testing, SBST@ICSE 2022, May 9, 2022. IEEE, Pittsburgh, PA, USA, pp 25\u201332. https:\/\/doi.org\/10.1145\/3526072.3527538","DOI":"10.1145\/3526072.3527538"},{"key":"10286_CR43","doi-asserted-by":"publisher","unstructured":"Gambi A, Mueller M, Fraser G (2019) AsFault: Testing self-driving car software using search-based procedural content generation. In: J.M. Atlee, T. Bultan, J. Whittle (eds.) 2019 IEEE\/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), pp. 27\u201330. IEEE, DOI https:\/\/doi.org\/10.1109\/icse-companion.2019.00030","DOI":"10.1109\/icse-companion.2019.00030"},{"key":"10286_CR44","doi-asserted-by":"publisher","unstructured":"Gambi A, M\u00fcller M., Fraser G (2019) Automatically testing self-driving cars with search-based procedural content generation. In: Zhang D, M\u00f8ller A (eds) Proceedings of the 28th ACM SIGSOFT international symposium on software testing and analysis, ISSTA 2019, July 15-19, 2019. ACM, Beijing, China, pp 318\u2013328. https:\/\/doi.org\/10.1145\/3293882.3330566","DOI":"10.1145\/3293882.3330566"},{"key":"10286_CR45","doi-asserted-by":"publisher","unstructured":"Gonz\u00e1lez CA, Varmazyar M, Nejati S, Briand LC, Isasi Y (2018) Enabling model testing of cyber-physical systems. In: Proceedings of the 21th ACM\/IEEE international conference on model driven engineering languages and systems, MODELS \u201918. Association for Computing Machinery, New York, NY, USA, pp 176\u2013186. https:\/\/doi.org\/10.1145\/3239372.3239409","DOI":"10.1145\/3239372.3239409"},{"key":"10286_CR46","unstructured":"Guardian T (2018) Self-driving uber kills arizona woman in first fatal crash involving pedestrian. https:\/\/www.theguardian.com\/technology\/2018\/mar\/19\/uber-self-driving-car-kills-woman-arizona-tempe. Accessed 8 Nov 2022"},{"key":"10286_CR47","doi-asserted-by":"publisher","unstructured":"Gundu R, Maleki M (2022) Securing CAN bus in connected and autonomous vehicles using supervised machine learning approaches. In: 2022 IEEE international conference on electro information technology, EIT 2022, May 19-21, 2022. IEEE, Mankato, MN, USA, pp 42\u201346. https:\/\/doi.org\/10.1109\/eIT53891.2022.9813985","DOI":"10.1109\/eIT53891.2022.9813985"},{"issue":"8","key":"10286_CR48","doi-asserted-by":"publisher","first-page":"832","DOI":"10.1109\/34.709601","volume":"20","author":"TK Ho","year":"1998","unstructured":"Ho TK (1998) The random subspace method for constructing decision forests. IEEE Trans Pattern Anal Mach Intell 20(8):832\u2013844. https:\/\/doi.org\/10.1109\/34.709601","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"10286_CR49","doi-asserted-by":"publisher","unstructured":"Huang J, Zhang C, Dolby J (2013) CLAP: recording local executions to reproduce concurrency failures. In: Boehm H, Flanagan C (eds) ACM SIGPLAN conference on programming language design and implementation, PLDI \u201913, June 16-19, 2013. ACM, Seattle, WA, USA, pp 141\u2013152. https:\/\/doi.org\/10.1145\/2491956.2462167","DOI":"10.1145\/2491956.2462167"},{"key":"10286_CR50","doi-asserted-by":"crossref","unstructured":"Ingrand F (2019) Recent trends in formal validation and verification of autonomous robots software. In: 3rd IEEE international conference on robotic computing, IRC 2019, February 25-27, 2019, Naples, Italy, pp 321\u2013328","DOI":"10.1109\/IRC.2019.00059"},{"issue":"10","key":"10286_CR51","doi-asserted-by":"publisher","first-page":"1379","DOI":"10.1016\/j.infsof.2009.04.016","volume":"51","author":"Y Jia","year":"2009","unstructured":"Jia Y, Harman M (2009) Higher order mutation testing. Inf Softw Technol 51(10):1379\u20131393","journal-title":"Inf Softw Technol"},{"key":"10286_CR52","doi-asserted-by":"publisher","first-page":"182","DOI":"10.1016\/j.tra.2016.09.010","volume":"94","author":"N Kalra","year":"2016","unstructured":"Kalra N, Paddock S (2016) Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability? Transp Res A Policy Pract 94:182\u2013193. https:\/\/doi.org\/10.1016\/j.tra.2016.09.010","journal-title":"Transp Res A Policy Pract"},{"key":"10286_CR53","doi-asserted-by":"crossref","unstructured":"Kaur A, Malhotra R (2008) Application of random forest in predicting fault-prone classes. In: 2008 international conference on advanced computer theory and engineering, pp 37\u201343","DOI":"10.1109\/ICACTE.2008.204"},{"key":"10286_CR54","doi-asserted-by":"publisher","unstructured":"Khatiri S, Birchler C, Bosshard B, Gambi A, Panichella S (2021) Machine learning-based test selection for simulation-based testing of self-driving cars software. https:\/\/doi.org\/10.5281\/zenodo.5085251","DOI":"10.5281\/zenodo.5085251"},{"key":"10286_CR55","doi-asserted-by":"publisher","unstructured":"Kim J, Chon S, Park J (2019) Suggestion of testing method for industrial level cyber-physical system in complex environment. In: 2019 IEEE international conference on software testing, verification and validation workshops (ICSTW). IEEE. https:\/\/doi.org\/10.1109\/icstw.2019.00043","DOI":"10.1109\/icstw.2019.00043"},{"key":"10286_CR56","doi-asserted-by":"crossref","unstructured":"Li G, Li Y, Jha S, Tsai T, Sullivan M, Hari SKS, Kalbarczyk Z, Iyer R (2020) AV-FUZZER: Finding safety violations in autonomous driving systems. In: 2020 IEEE 31st international symposium on software reliability engineering (ISSRE). IEEE, pp 25\u201336","DOI":"10.1109\/ISSRE5003.2020.00012"},{"issue":"2","key":"10286_CR57","doi-asserted-by":"publisher","first-page":"158","DOI":"10.1109\/TIV.2016.2608003","volume":"1","author":"L Li","year":"2016","unstructured":"Li L, Huang W, Liu Y, Zheng N, Wang F (2016) Intelligence testing for autonomous vehicles: A new approach. IEEE Trans Intell Veh 1(2):158\u2013166","journal-title":"IEEE Trans Intell Veh"},{"key":"10286_CR58","unstructured":"Ling CX, Li C (1998) Data mining for direct marketing: Problems and solutions. In: Proceedings of the 4th international conference on knowledge discovery and data mining, KDD\u201998. AAAI Press, pp 73\u201379"},{"issue":"1","key":"10286_CR59","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/TRO.2019.2942989","volume":"36","author":"A Loquercio","year":"2020","unstructured":"Loquercio A, Kaufmann E, Ranftl R, Dosovitskiy A, Koltun V, Scaramuzza D (2020) Deep drone racing: From simulation to reality with domain randomization. IEEE Trans Robot 36(1):1\u201314","journal-title":"IEEE Trans Robot"},{"key":"10286_CR60","doi-asserted-by":"crossref","unstructured":"Matinnejad R, Nejati S, Briand L, Bruckmann T, Poull C (2013) Automated model-in-the-loop testing of continuous controllers using search. In: International symposium on search based software engineering. Springer, pp 141\u2013157","DOI":"10.1007\/978-3-642-39742-4_12"},{"issue":"6","key":"10286_CR61","doi-asserted-by":"publisher","first-page":"674","DOI":"10.1109\/TSE.2018.2868082","volume":"46","author":"DD Nucci","year":"2020","unstructured":"Nucci DD, Panichella A, Zaidman A, Lucia AD (2020) A test case prioritization genetic algorithm guided by the hypervolume indicator. IEEE Trans Software Eng 46(6):674\u2013696. https:\/\/doi.org\/10.1109\/TSE.2018.2868082","journal-title":"IEEE Trans Software Eng"},{"key":"10286_CR62","unstructured":"Nvidia (2020) Nvidia drive constellation. https:\/\/developer.nvidia.com\/drive\/drive-constellation. Accessed 8 Nov 2022"},{"issue":"2","key":"10286_CR63","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1007\/s10664-021-10066-6","volume":"27","author":"R Pan","year":"2022","unstructured":"Pan R, Bagherzadeh M, Ghaleb TA, Briand LC (2022) Test case selection and prioritization using machine learning: a systematic literature review. Empir Softw Eng 27(2):29. https:\/\/doi.org\/10.1007\/s10664-021-10066-6","journal-title":"Empir Softw Eng"},{"key":"10286_CR64","doi-asserted-by":"publisher","unstructured":"Panichella S (2015) Supporting newcomers in software development projects. In: 2015 IEEE international conference on software maintenance and evolution (ICSME), pp 586\u2013589. https:\/\/doi.org\/10.1109\/ICSM.2015.7332519","DOI":"10.1109\/ICSM.2015.7332519"},{"key":"10286_CR65","doi-asserted-by":"crossref","unstructured":"Panichella S, Gambi A, Zampetti F, Riccio V (2021) SBST tool competition 2021. In: International conference on software engineering, workshops. ACM, Madrid, Spain","DOI":"10.1109\/SBST52555.2021.00011"},{"key":"10286_CR66","doi-asserted-by":"publisher","unstructured":"Panichella S, Ruiz M (2020) Requirements-collector: Automating requirements specification from elicitation sessions and user feedback. In: Breaux TD, Zisman A, Fricker S, Glinz M (eds) 28th IEEE international requirements engineering conference, RE 2020, August 31 - September 4, 2020. IEEE, Zurich, Switzerland, pp 404\u2013407. https:\/\/doi.org\/10.1109\/RE48521.2020.00057","DOI":"10.1109\/RE48521.2020.00057"},{"key":"10286_CR67","doi-asserted-by":"publisher","unstructured":"Panichella S, Sorbo AD, Guzman E, Visaggio CA, Canfora G, Gall HC (2015) How can I improve my app? classifying user reviews for software maintenance and evolution. In: Koschke R, Krinke J, Robillard MP (eds) International conference on software maintenance and evolution, ICSME. IEEE Computer Society, pp 281\u2013290. https:\/\/doi.org\/10.1109\/ICSM.2015.7332474","DOI":"10.1109\/ICSM.2015.7332474"},{"key":"10286_CR68","doi-asserted-by":"publisher","first-page":"111047","DOI":"10.1016\/j.jss.2021.111047","volume":"181","author":"P Rani","year":"2021","unstructured":"Rani P, Panichella S, Leuenberger M, Di Sorbo A, Nierstrasz O (2021) How to identify class comment types? A multi-language approach for class comment classification. J Syst Softw 181:111047. https:\/\/doi.org\/10.1016\/j.jss.2021.111047, https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0164121221001448","journal-title":"J Syst Softw"},{"key":"10286_CR69","doi-asserted-by":"publisher","first-page":"532","DOI":"10.1007\/978-0-387-39940-9_565","volume-title":"Cross-validation","author":"P Refaeilzadeh","year":"2009","unstructured":"Refaeilzadeh P, Tang L, Liu H (2009) Cross-validation. Springer US, Boston, pp 532\u2013538. https:\/\/doi.org\/10.1007\/978-0-387-39940-9_565"},{"key":"10286_CR70","doi-asserted-by":"publisher","unstructured":"Riccio V, Tonella P (2020) Model-based exploration of the frontier of behaviours for deep learning system testing. In: Proceedings of the ACM joint european software engineering conference and symposium on the foundations of software engineering, ESEC\/FSE \u201920. Association for Computing Machinery, p 13. https:\/\/doi.org\/10.1145\/3368089.3409730","DOI":"10.1145\/3368089.3409730"},{"key":"10286_CR71","doi-asserted-by":"publisher","unstructured":"Roper M (2019) Using machine learning to classify test outcomes. In: IEEE international conference on artificial intelligence testing, AITest 2019, April 4-9, 2019. IEEE, Newark, CA, USA, pp 99\u2013100. https:\/\/doi.org\/10.1109\/AITest.2019.00009","DOI":"10.1109\/AITest.2019.00009"},{"key":"10286_CR72","doi-asserted-by":"crossref","unstructured":"Rothermel G, Harrold MJ, Ostrin J, Hong C (1998) An empirical study of the effects of minimization on the fault detection capabilities of test suites. In: Proceedings of the international conference on software maintenance. IEEE CS Press, pp 34\u201344","DOI":"10.1109\/ICSM.1998.738487"},{"key":"10286_CR73","doi-asserted-by":"publisher","unstructured":"Rothermel G, Untch R, Chu C, Harrold M (1999) Test case prioritization: an empirical study. In: IEEE international conference on software maintenance, 1999. (ICSM \u201999) Proceedings. IEEE, pp 179\u2013188. https:\/\/doi.org\/10.1109\/ICSM.1999.792604","DOI":"10.1109\/ICSM.1999.792604"},{"key":"10286_CR74","doi-asserted-by":"publisher","unstructured":"Sadri-Moshkenani Z, Bradley JM, Rothermel G (2022) Survey on test case generation, selection and prioritization for cyber-physical systems. Softw Test Verification Reliab 32(1). https:\/\/doi.org\/10.1002\/stvr.1794","DOI":"10.1002\/stvr.1794"},{"key":"10286_CR75","doi-asserted-by":"publisher","unstructured":"Sammut C, Webb GI (eds) (2011) Logistic regression. Springer US, Boston. https:\/\/doi.org\/10.1007\/978-0-387-30164-8_493","DOI":"10.1007\/978-0-387-30164-8_493"},{"key":"10286_CR76","unstructured":"Sean CG (2022) Casper van der Wel. 2007-2022, S.C.: Shapely. https:\/\/github.com\/shapely\/shapely. Accessed 8 Nov 2022"},{"key":"10286_CR77","doi-asserted-by":"publisher","unstructured":"Shin SY, Nejati S, Sabetzadeh M, Briand LC, Zimmer F (2018) Test case prioritization for acceptance testing of cyber physical systems: a multi-objective search-based approach. In: Tip F, Bodden E (eds) Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis, ISSTA 2018, July 16-21, 2018. ACM, Amsterdam, The Netherlands, pp 49\u201360. https:\/\/doi.org\/10.1145\/3213846.3213852","DOI":"10.1145\/3213846.3213852"},{"issue":"6","key":"10286_CR78","doi-asserted-by":"publisher","first-page":"1855","DOI":"10.1109\/TITS.2017.2742141","volume":"19","author":"S Sontges","year":"2018","unstructured":"Sontges S, Althoff M (2018) Computing the drivable area of autonomous road vehicles in dynamic road scenes. IEEE Trans Intell Trans Syst 19 (6):1855\u20131866. https:\/\/doi.org\/10.1109\/TITS.2017.2742141","journal-title":"IEEE Trans Intell Trans Syst"},{"key":"10286_CR79","doi-asserted-by":"publisher","unstructured":"Sorbo AD, Panichella S, Alexandru CV, Shimagaki J, Visaggio CA, Canfora G, Gall HC (2016) What would users change in my app? summarizing app reviews for recommending software changes. In: Zimmermann T, Cleland-Huang J, Su Z (eds) International symposium on foundations of software engineering. ACM, pp 499\u2013510. https:\/\/doi.org\/10.1145\/2950290.2950299","DOI":"10.1145\/2950290.2950299"},{"key":"10286_CR80","unstructured":"Sorbo AD, Zampetti F, Visaggio CA, Penta MD, Panichella S (2022) Automated identification and qualitative characterization of safety concerns reported in UAV software platforms. Trans Softw Eng Methodol"},{"key":"10286_CR81","unstructured":"The-Washington-Post (2019) Uber\u2019s radar detected Elaine Herzberg nearly 6 seconds before she was fatally struck but \u201cthe system design did not include a consideration for jaywalking pedestrians\u201d so it didn\u2019t react as if she were a person. https:\/\/mobile.twitter.com\/faizsays\/status\/1191885955088519168"},{"issue":"5","key":"10286_CR82","doi-asserted-by":"publisher","first-page":"533","DOI":"10.1001\/jama.2016.7653","volume":"316","author":"J Tolles","year":"2016","unstructured":"Tolles J, Meurer WJ (2016) Logistic regression. JAMA 316 (5):533. https:\/\/doi.org\/10.1001\/jama.2016.7653","journal-title":"JAMA"},{"key":"10286_CR83","doi-asserted-by":"publisher","unstructured":"Xu J, Luo Q, Xu K, Xiao X, Yu S, Hu J, Miao J, Wang J (2019) An automated learning-based procedure for large-scale vehicle dynamics modeling on Baidu Apollo platform. In: 2019 IEEE\/RSJ international conference on intelligent robots and systems, IROS. IEEE, pp 5049\u20135056. https:\/\/doi.org\/10.1109\/IROS40897.2019.8968102","DOI":"10.1109\/IROS40897.2019.8968102"},{"issue":"4","key":"10286_CR84","doi-asserted-by":"publisher","first-page":"689","DOI":"10.1016\/j.jss.2009.11.706","volume":"83","author":"S Yoo","year":"2010","unstructured":"Yoo S, Harman M (2010) Using hybrid algorithm for Pareto efficient multi-objective test suite minimisation. J Syst Softw 83(4):689\u2013701","journal-title":"J Syst Softw"},{"issue":"2","key":"10286_CR85","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1002\/stv.430","volume":"22","author":"S Yoo","year":"2012","unstructured":"Yoo S, Harman M (2012) Regression testing minimization, selection and prioritization: a survey. Softw Test Verification Reliab 22(2):67\u2013120","journal-title":"Softw Test Verification Reliab"},{"key":"10286_CR86","doi-asserted-by":"crossref","unstructured":"Zapridou E, Bartocci E, Katsaros P Deshmukh J, Ni\u010dkovi\u0107 D (eds) (2020) Runtime verification of autonomous driving systems in Carla. Springer International Publishing, Cham","DOI":"10.1007\/978-3-030-60508-7_9"},{"key":"10286_CR87","doi-asserted-by":"crossref","unstructured":"Zhang XY, Arcaini P, Ishikawa F, Liu K (2020) Investigating the configurations of an industrial path planner in terms of collision avoidance. In: 2020 IEEE 31st international symposium on software reliability engineering (ISSRE), pp 301\u2013312. https:\/\/doi.org\/10.1109\/ISSRE5003.2020.00036","DOI":"10.1109\/ISSRE5003.2020.00036"},{"issue":"5","key":"10286_CR88","doi-asserted-by":"publisher","first-page":"618","DOI":"10.1109\/TSE.2010.63","volume":"36","author":"T Zimmermann","year":"2010","unstructured":"Zimmermann T, Premraj R, Bettenburg N, Just S, Schr\u00f6ter A, Weiss C (2010) What makes a good bug report. IEEE Trans Software Eng 36 (5):618\u2013643. https:\/\/doi.org\/10.1109\/TSE.2010.63","journal-title":"IEEE Trans Software Eng"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-023-10286-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-023-10286-y\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-023-10286-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,5,26]],"date-time":"2023-05-26T09:48:59Z","timestamp":1685094539000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-023-10286-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,26]]},"references-count":88,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,5]]}},"alternative-id":["10286"],"URL":"https:\/\/doi.org\/10.1007\/s10664-023-10286-y","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,4,26]]},"assertion":[{"value":"20 December 2022","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 April 2023","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declared that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"<!--Emphasis Type='Bold' removed-->Conflict of Interests"}}],"article-number":"71"}}