{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,17]],"date-time":"2026-07-17T15:27:07Z","timestamp":1784302027984,"version":"3.55.0"},"reference-count":93,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2025,9,25]],"date-time":"2025-09-25T00:00:00Z","timestamp":1758758400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,9,25]],"date-time":"2025-09-25T00:00:00Z","timestamp":1758758400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"University of Bern"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2025,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Despite the recent developments in obstacle avoidance and other safety features, autonomous Unmanned Aerial Vehicles (UAVs) continue to face safety challenges. No previous work investigated the relationship between the behavioral uncertainty of a UAV, characterized in this work by inconsistent or erratic control signal patterns, and the unsafety of its flight. By quantifying uncertainty, it is possible to develop a predictor for unsafety, which acts as a flight supervisor. We conducted a large-scale empirical investigation of safety violations using PX4-Autopilot, an open-source UAV software platform. Our dataset of over 5,000 simulated flights, created to challenge obstacle avoidance, allowed us to explore the relation between uncertain UAV decisions and safety violations: up to 89% of unsafe UAV states exhibit significant decision uncertainty, and up to 74% of uncertain decisions lead to unsafe states. Based on these findings, we implemented\n                    <jats:sc>Superialist<\/jats:sc>\n                    (Supervising Autonomous Aerial Vehicles), a\n                    <jats:italic>runtime uncertainty detector<\/jats:italic>\n                    based on autoencoders, the state-of-the-art technology for anomaly detection.\n                    <jats:sc>Superialist<\/jats:sc>\n                    achieved high performance in detecting uncertain behaviors with up to 96% precision and 93% recall. Despite the observed performance degradation when using the same approach for predicting unsafety (up to 74% precision and 87% recall),\n                    <jats:sc>Superialist<\/jats:sc>\n                    enabled early prediction of unsafe states up to 50 seconds in advance.\n                  <\/jats:p>","DOI":"10.1007\/s10664-025-10697-z","type":"journal-article","created":{"date-parts":[[2025,9,25]],"date-time":"2025-09-25T13:55:30Z","timestamp":1758808530000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":17,"title":["When uncertainty leads to unsafety: Empirical insights into the role of uncertainty in unmanned aerial vehicle safety"],"prefix":"10.1007","volume":"30","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0354-9747","authenticated-orcid":false,"given":"Sajad","family":"Khatiri","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7432-437X","authenticated-orcid":false,"given":"Fatemeh","family":"Mohammadi Amin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4120-626X","authenticated-orcid":false,"given":"Sebastiano","family":"Panichella","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3088-0339","authenticated-orcid":false,"given":"Paolo","family":"Tonella","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,9,25]]},"reference":[{"key":"10697_CR1","doi-asserted-by":"publisher","unstructured":"Abdessalem RB, Nejati S, Briand LC, Stifter T (2018) Testing vision-based control systems using learnable evolutionary algorithms. In: International conference on software engineering. ACM, pp 1016\u20131026. https:\/\/doi.org\/10.1145\/3180155.3180160","DOI":"10.1145\/3180155.3180160"},{"key":"10697_CR2","doi-asserted-by":"crossref","unstructured":"Afzal A, Katz DS, Le\u00a0Goues C, Timperley CS (2021) Simulation for robotics test automation: Developer perspectives. In: Conference on software testing, verification and validation. IEEE, pp 263\u2013274","DOI":"10.1109\/ICST49551.2021.00036"},{"key":"10697_CR3","doi-asserted-by":"crossref","unstructured":"Afzal A, Le\u00a0Goues C, Hilton M, Timperley CS (2020) A study on challenges of testing robotic systems. In: International conference on software testing, validation and verification. IEEE, pp 96\u2013107","DOI":"10.1109\/ICST46399.2020.00020"},{"key":"10697_CR4","doi-asserted-by":"crossref","unstructured":"Afzal A, Le\u00a0Goues C, Timperley CS (2021) Mithra: anomaly detection as an oracle for cyberphysical systems. IEEE Trans Softw Eng","DOI":"10.1109\/TSE.2021.3120680"},{"key":"10697_CR5","doi-asserted-by":"crossref","unstructured":"Ahn H (2020) Deep learning based anomaly detection for a vehicle in swarm drone system. In: 2020 International conference on unmanned aircraft systems (ICUAS). IEEE, pp 557\u2013561","DOI":"10.1109\/ICUAS48674.2020.9213880"},{"key":"10697_CR6","doi-asserted-by":"publisher","unstructured":"Alcon M, Tabani H, Abella J, Cazorla FJ (2021) Enabling unit testing of already-integrated AI software systems: the case of apollo for autonomous driving. In: Euromicro conference on digital system design. IEEE, pp 426\u2013433. https:\/\/doi.org\/10.1109\/DSD53832.2021.00071","DOI":"10.1109\/DSD53832.2021.00071"},{"issue":"2","key":"10697_CR7","doi-asserted-by":"publisher","first-page":"47","DOI":"10.1007\/S10664-023-10433-5","volume":"29","author":"MH Amini","year":"2024","unstructured":"Amini MH, Naseri S, Nejati S (2024) Evaluating the impact of flaky simulators on testing autonomous driving systems. Empir Softw Eng 29(2):47. https:\/\/doi.org\/10.1007\/S10664-023-10433-5","journal-title":"Empir Softw Eng"},{"key":"10697_CR8","unstructured":"Ardupilot.org (2007) Ardupilot \u2013 versatile, trusted, open. https:\/\/ardupilot.org\/. Accessed: 02-July-2022"},{"key":"10697_CR9","unstructured":"Arnez F, Espinoza H, Radermacher A, Terrier F (2020) A comparison of uncertainty estimation approaches in deep learning components for autonomous vehicle applications. arXiv:2006.15172"},{"key":"10697_CR10","doi-asserted-by":"crossref","unstructured":"Arnez F, Espinoza H, Radermacher A, Terrier F (2022) Towards dependable autonomous systems based on bayesian deep learning components. In: 2022 18th European dependable computing conference (EDCC). IEEE, pp 65\u201372","DOI":"10.1109\/EDCC57035.2022.00021"},{"key":"10697_CR11","doi-asserted-by":"publisher","unstructured":"Asmat MN, ur\u00a0Rehman\u00a0Khan S, Hussain S (2023) Uncertainty handling in cyber-physical systems: State-of-the-art approaches, tools, causes, and future directions. J Softw Evol Process 35(7). https:\/\/doi.org\/10.1002\/smr.2428","DOI":"10.1002\/smr.2428"},{"issue":"24","key":"10697_CR12","doi-asserted-by":"publisher","first-page":"8253","DOI":"10.3390\/s21248253","volume":"21","author":"E Balestrieri","year":"2021","unstructured":"Balestrieri E, Daponte P, De Vito L, Picariello F, Tudosa I (2021) Sensors and measurements for UAV safety: an overview. Sensors 21(24):8253. https:\/\/doi.org\/10.3390\/s21248253","journal-title":"Sensors"},{"key":"10697_CR13","unstructured":"Baumann T (2018) Obstacle avoidance for drones using a 3dvfh* algorithm. Spring Term 2018, 67"},{"key":"10697_CR14","doi-asserted-by":"crossref","unstructured":"Bektash O, la\u00a0Cour-Harbo A (2020) Vibration analysis for anomaly detection in unmanned aircraft. In: Annual conference of the prognostics and health management society 2020. PHM Society","DOI":"10.36001\/phmconf.2020.v12i1.1143"},{"key":"10697_CR15","unstructured":"Berndt DJ, Clifford J (1994) Using dynamic time warping to find patterns in time series. In: KDD workshop, 16. Seattle, WA, USA, pp 359\u2013370"},{"key":"10697_CR16","doi-asserted-by":"crossref","unstructured":"Birchler C, Ganz N, Khatiri S, Gambi A, Panichella S (2022) Cost-effective simulation-based test selection in self-driving cars software. Sci Comput Program (SCP)","DOI":"10.1016\/j.scico.2023.102926"},{"key":"10697_CR17","doi-asserted-by":"crossref","unstructured":"Birchler C, Ganz N, Khatiri S, Gambi A, Panichella S (2022) Cost-effective simulation-based test selection in self-driving cars software with sdc-scissor. In: the 29th IEEE international conference on software analysis, evolution, and reengineering","DOI":"10.1109\/SANER53432.2022.00030"},{"key":"10697_CR18","doi-asserted-by":"crossref","unstructured":"Birchler C, Khatiri S, Bosshard B, Gambi A, Panichella S (2022) Machine learning-based test selection for simulation-based testing of self-driving cars software. Empir Softw Eng","DOI":"10.1007\/s10664-023-10286-y"},{"key":"10697_CR19","doi-asserted-by":"crossref","unstructured":"Birchler C, Khatiri S, Derakhshanfar P, Panichella S, Panichella A (2022) Single and multi-objective test cases prioritization for self-driving cars in virtual environments. ACM Trans Softw Eng Methodol (TOSEM)","DOI":"10.1145\/3533818"},{"key":"10697_CR20","doi-asserted-by":"publisher","DOI":"10.1145\/3711906","author":"C Birchler","year":"2025","unstructured":"Birchler C, Khatiri S, Rani P, Kehrer T, Panichella S (2025) A roadmap for simulation-based testing of autonomous cyber-physical systems: challenges and future direction. ACM Trans Softw Eng Methodol. https:\/\/doi.org\/10.1145\/3711906","journal-title":"ACM Trans Softw Eng Methodol"},{"key":"10697_CR21","doi-asserted-by":"crossref","unstructured":"Birchler C, Mohammed TK, Rani P, Nechita T, Kehrer T, Panichella S (2024) How does simulation-based testing for self-driving cars match human perception? In: ACM international conference on the foundations of software engineering","DOI":"10.1145\/3643768"},{"key":"10697_CR22","doi-asserted-by":"crossref","unstructured":"Birchler C, Rohrbach C, Kim H, Gambi A, Liu T, Horneber J, Kehrer T, Panichella S (2023) TEASER: simulation-based can bus regression testing for self-driving cars software. IEEE\/ACM Int Conf Autom Softw Eng","DOI":"10.1109\/ASE56229.2023.00154"},{"key":"10697_CR23","doi-asserted-by":"publisher","unstructured":"Bojarczuk K, Gucevska N, Lucas SMM, Dvortsova I, Harman M, Meijer E, Sapora S, George J, Lomeli M, Rojas R (2021) Measurement challenges for cyber cyber digital twins: Experiences from the deployment of facebook\u2019s WW simulation system. In: International symposium on empirical software engineering and measurement. ACM, pp 2:1\u20132:10. https:\/\/doi.org\/10.1145\/3475716.3484196","DOI":"10.1145\/3475716.3484196"},{"key":"10697_CR24","doi-asserted-by":"publisher","unstructured":"Carbone C, Albani D, Magistri F, Ognibene D, Stachniss C, Kootstra G, Nardi D, Trianni V (2021) Monitoring and mapping of crop fields with UAV swarms based on information gain. In: Distributed autonomous robotic systems - international symposium, Springer Proceedings in Advanced Robotics. Springer, vol\u00a022, pp 306\u2013319. https:\/\/doi.org\/10.1007\/978-3-030-92790-5_24","DOI":"10.1007\/978-3-030-92790-5_24"},{"issue":"4","key":"10697_CR25","doi-asserted-by":"publisher","first-page":"79:1","DOI":"10.1145\/3527451","volume":"31","author":"F\u00d6 \u00c7atak","year":"2022","unstructured":"\u00c7atak F\u00d6, Yue T, Ali S (2022) Uncertainty-aware prediction validator in deep learning models for cyber-physical system data. ACM Trans Softw Eng Methodol 31(4):79:1-79:31. https:\/\/doi.org\/10.1145\/3527451","journal-title":"ACM Trans Softw Eng Methodol"},{"key":"10697_CR26","doi-asserted-by":"publisher","first-page":"107495","DOI":"10.1016\/j.cie.2021.107495","volume":"159","author":"S Chowdhury","year":"2021","unstructured":"Chowdhury S, Shahvari O, Marufuzzaman M, Li X, Bian L (2021) Drone routing and optimization for post-disaster inspection. Comput Ind Eng 159:107495","journal-title":"Comput Ind Eng"},{"issue":"3","key":"10697_CR27","doi-asserted-by":"publisher","first-page":"647","DOI":"10.1109\/TASE.2014.2326952","volume":"11","author":"R D\u2019Andrea","year":"2014","unstructured":"D\u2019Andrea R (2014) Guest editorial can drones deliver? IEEE Trans Autom Sci Eng 11(3):647\u2013648. https:\/\/doi.org\/10.1109\/TASE.2014.2326952","journal-title":"IEEE Trans Autom Sci Eng"},{"key":"10697_CR28","doi-asserted-by":"crossref","unstructured":"Denney E, Pai G, Habli I (2015) Dynamic safety cases for through-life safety assurance. In: 2015 IEEE\/ACM 37th IEEE international conference on software engineering. IEEE, vol\u00a02, pp 587\u2013590","DOI":"10.1109\/ICSE.2015.199"},{"key":"10697_CR29","doi-asserted-by":"crossref","unstructured":"Di Sorbo A, Zampetti F, Visaggio CA, Penta MD, Panichella S (2022) Automated identification and qualitative characterization of safety concerns reported in uav software platforms. Trans Softw Eng Methodol","DOI":"10.1145\/3564821"},{"key":"10697_CR30","unstructured":"EUDronePort (2025) C1 drones: certification requirements and operational guidelines. Blog Post. https:\/\/eudroneport.com\/blog\/certification-requirements-c1-drones\/"},{"key":"#cr-split#-10697_CR31.1","unstructured":"European Commission (2019) Commission delegated regulation"},{"key":"#cr-split#-10697_CR31.2","unstructured":"(EU) 2019\/945 of 12 March 2019 on unmanned aircraft systems and on third-country operators of unmanned aircraft systems. EU Regulation L 152\/1, Official Journal of the European Union, Brussels. Defines drone classes C0-C6 and technical requirements"},{"key":"#cr-split#-10697_CR32.1","unstructured":"European Commission (2019) Commission Implementing Regulation"},{"key":"#cr-split#-10697_CR32.2","unstructured":"(EU) 2019\/947 of 24 May 2019 on the rules and procedures for the operation of unmanned aircraft. EU Regulation L 152\/45, Official Journal of the European Union, Brussels. Defines operational categories. Open, Specific, Certified"},{"key":"10697_CR33","unstructured":"European Parliament and Council of the European Union (2014) Directive 2014\/53\/EU of the European Parliament and of the Council of 16 April 2014 on the harmonisation of the laws of the Member States relating to the making available on the market of radio equipment. Official Journal of the European Union. Radio Equipment Directive (RED). Relevant for CE marking of drones"},{"key":"10697_CR34","doi-asserted-by":"publisher","unstructured":"Galvan J, Raja A, Li Y, Yuan J (2021) Sensor data-driven uav anomaly detection using deep learning approach. In: MILCOM 2021 - 2021 IEEE military communications conference (MILCOM), pp 589\u2013594. https:\/\/doi.org\/10.1109\/MILCOM52596.2021.9653036","DOI":"10.1109\/MILCOM52596.2021.9653036"},{"key":"10697_CR35","doi-asserted-by":"crossref","unstructured":"Garcia J, Feng Y, Shen J, Almanee S, Xia Y, Chen QA (2020) A comprehensive study of autonomous vehicle bugs. In: ICSE \u201920: 42nd international conference on software engineering, Seoul, South Korea, 27 June - 19 July, 2020. ACM, pp 385\u2013396","DOI":"10.1145\/3377811.3380397"},{"key":"10697_CR36","doi-asserted-by":"crossref","unstructured":"Garc\u00eda J, Molina JM (2020) Simulation in real conditions of navigation and obstacle avoidance with px4\/gazebo platform. Person Ubiquit Comput 1\u201321","DOI":"10.1007\/s00779-019-01356-4"},{"issue":"4","key":"10697_CR37","doi-asserted-by":"publisher","first-page":"95:1","DOI":"10.1145\/3576041","volume":"32","author":"L Han","year":"2023","unstructured":"Han L, Ali S, Yue T, Arrieta A, Arratibel M (2023) Uncertainty-aware robustness assessment of industrial elevator systems. ACM Trans Softw Eng Methodol 32(4):95:1-95:51. https:\/\/doi.org\/10.1145\/3576041","journal-title":"ACM Trans Softw Eng Methodol"},{"issue":"1","key":"10697_CR38","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1109\/TASE.2021.3073396","volume":"20","author":"W Hao","year":"2023","unstructured":"Hao W, Yang T, Yang Q (2023) Hybrid statistical-machine learning for real-time anomaly detection in industrial cyber-physical systems. IEEE Trans Autom Sci Eng 20(1):32\u201346. https:\/\/doi.org\/10.1109\/TASE.2021.3073396","journal-title":"IEEE Trans Autom Sci Eng"},{"key":"10697_CR39","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1016\/j.ssci.2013.04.007","volume":"59","author":"R Hawkins","year":"2013","unstructured":"Hawkins R, Habli I, Kelly T, McDermid J (2013) Assurance cases and prescriptive software safety certification: a comparative study. Safety Sci 59:55\u201371","journal-title":"Safety Sci"},{"key":"10697_CR40","doi-asserted-by":"crossref","unstructured":"Hildebrandt C, Elbaum S (2021) World-in-the-loop simulation for autonomous systems validation. In: 2021 IEEE international conference on robotics and automation (ICRA). IEEE, pp 10912\u201310919","DOI":"10.1109\/ICRA48506.2021.9561240"},{"issue":"19","key":"10697_CR41","doi-asserted-by":"publisher","first-page":"6340","DOI":"10.3390\/s21196340","volume":"21","author":"Z Huang","year":"2021","unstructured":"Huang Z, Shen Y, Li J, Fey M, Brecher C (2021) A survey on ai-driven digital twins in industry 4.0: smart manufacturing and advanced robotics. Sensors 21(19):6340. https:\/\/doi.org\/10.3390\/s21196340","journal-title":"Sensors"},{"key":"10697_CR42","unstructured":"Kendall A, Gal Y (2017) What uncertainties do we need in bayesian deep learning for computer vision? Adv Neural Inf Process Syst 30"},{"key":"10697_CR43","unstructured":"keras.io (2020) Timeseries anomaly detection using an autoencoder. https:\/\/keras.io\/examples\/timeseries\/timeseries_anomaly_detection\/. Accessed: 04-March-2024"},{"key":"10697_CR44","doi-asserted-by":"publisher","unstructured":"Khatiri S, Mohammadi Amin F, Panichella S, Tonella P (2023) Replication package: \u201cwhen uncertainty leads to unsafety: empirical insights into the role of uncertainty in unmanned aerial vehicle safety\u201d. https:\/\/doi.org\/10.5281\/zenodo.10393522","DOI":"10.5281\/zenodo.10393522"},{"key":"10697_CR45","doi-asserted-by":"crossref","unstructured":"Khatiri S, Panichella S, Tonella P (2023) Simulation-based test case generation for unmanned aerial vehicles in the neighborhood of real flights. In: International conference on software testing, verification and validation. IEEE, pp 281\u2013292. https:\/\/doi.org\/10.1109\/ICST57152.2023.00034","DOI":"10.1109\/ICST57152.2023.00034"},{"key":"10697_CR46","doi-asserted-by":"crossref","unstructured":"Khatiri S, Panichella S, Tonella P (2024) Simulation-based testing of unmanned aerial vehicles with aerialist. In: 46th IEEE\/ACM international conference on software engineering (ICSE), Lisbon, Portugal, 14-20 April 2024","DOI":"10.1145\/3639478.3640031"},{"key":"10697_CR47","doi-asserted-by":"crossref","unstructured":"Khatiri S, Saurabh P, Zimmermann T, Munasinghe C, Birchler C, Panichella S (2024) Sbft tool competition 2024: Cps-uav test case generation track. In: 17th International workshop on search-based and fuzz testing (SBFT), Lisbon, Portugal, 14-20 April 2024","DOI":"10.1145\/3643659.3643931"},{"key":"10697_CR48","doi-asserted-by":"crossref","unstructured":"Khatiri S, Zohdinasab T, Saurabh P, Humeniuk D, Panichella S (2025) ICST tool competition 2025 - uav testing track. In: IEEE\/ACM international conference on software testing, verification and validation, ICST 2025","DOI":"10.1109\/ICST62969.2025.10988922"},{"key":"10697_CR49","doi-asserted-by":"crossref","unstructured":"Khatiri S, Zohdinasab T, Saurabh P, Humeniuk D, Panichella S (2025) SBFT tool competition 2025 - uav testing track. In: IEEE\/ACM international workshop on search-based and fuzz testing, SBFT@ICSE 2025","DOI":"10.1109\/SBFT66712.2025.00007"},{"key":"10697_CR50","doi-asserted-by":"publisher","unstructured":"Koenig NP, Howard A (2004) Design and use paradigms for gazebo, an open-source multi-robot simulator. In: International conference on intelligent robots and systems. IEEE, pp 2149\u20132154. https:\/\/doi.org\/10.1109\/IROS.2004.1389727","DOI":"10.1109\/IROS.2004.1389727"},{"issue":"3","key":"10697_CR51","doi-asserted-by":"publisher","first-page":"58:1","DOI":"10.1145\/3550270","volume":"32","author":"T Laurent","year":"2023","unstructured":"Laurent T, Klikovits S, Arcaini P, Ishikawa F, Ventresque A (2023) Parameter coverage for testing of autonomous driving systems under uncertainty. ACM Trans Softw Eng Methodol 32(3):58:1-58:31. https:\/\/doi.org\/10.1145\/3550270","journal-title":"ACM Trans Softw Eng Methodol"},{"key":"10697_CR52","doi-asserted-by":"crossref","unstructured":"Lindvall M, Porter A, Magnusson G, Schulze C (2017) Metamorphic model-based testing of autonomous systems. In: International workshop on metamorphic testing. IEEE, pp 35\u201341","DOI":"10.1109\/MET.2017.6"},{"issue":"4","key":"10697_CR53","first-page":"2315","volume":"5","author":"H Lu","year":"2017","unstructured":"Lu H, Li Y, Mu S, Wang D, Kim H, Serikawa S (2017) Motor anomaly detection for unmanned aerial vehicles using reinforcement learning. IEEE Int Things J 5(4):2315\u20132322","journal-title":"IEEE Int Things J"},{"key":"10697_CR54","doi-asserted-by":"crossref","unstructured":"Luo Q, Hariri F, Eloussi L, Marinov D (2014) An empirical analysis of flaky tests. In: Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering, (FSE-22), Hong Kong, China, November 16 - 22, 2014, pp 643\u2013653","DOI":"10.1145\/2635868.2635920"},{"issue":"3","key":"10697_CR55","doi-asserted-by":"publisher","first-page":"52","DOI":"10.1007\/s10664-021-09941-z","volume":"26","author":"T Ma","year":"2021","unstructured":"Ma T, Ali S, Yue T (2021) Testing self-healing cyber-physical systems under uncertainty with reinforcement learning: an empirical study. Empir Softw Eng 26(3):52. https:\/\/doi.org\/10.1007\/s10664-021-09941-z","journal-title":"Empir Softw Eng"},{"key":"10697_CR56","unstructured":"Matternet (2023) Faa grants approval to matternet for single pilot to operate up to 20 drones at test site. Unmanned Publications. https:\/\/www.unmannedairspace.info\/latest-news-and-information\/faa-grants-approval-for-matternet-for-single-pilot-to-operate-up-to-20-drones-at-test-site\/. Accessed: 16-Dec-2023"},{"key":"10697_CR57","doi-asserted-by":"crossref","unstructured":"McAllister RT, Gal Y, Kendall A, Van Der\u00a0Wilk M, Shah A, Cipolla R, Weller A (2017) Concrete problems for autonomous vehicle safety: Advantages of bayesian deep learning. In: International joint conferences on artificial intelligence, Inc","DOI":"10.24963\/ijcai.2017\/661"},{"key":"10697_CR58","doi-asserted-by":"crossref","unstructured":"Meier L, Honegger D, Pollefeys M (2015) Px4: a node-based multithreaded open source robotics framework for deeply embedded platforms. In: international conference on robotics and automation. IEEE, pp 6235\u20136240","DOI":"10.1109\/ICRA.2015.7140074"},{"key":"10697_CR59","doi-asserted-by":"crossref","unstructured":"Michelmore R, Wicker M, Laurenti L, Cardelli L, Gal Y, Kwiatkowska M (2020) Uncertainty quantification with statistical guarantees in end-to-end autonomous driving control. In: 2020 IEEE international conference on robotics and automation (ICRA). IEEE, pp 7344\u20137350","DOI":"10.1109\/ICRA40945.2020.9196844"},{"key":"10697_CR60","doi-asserted-by":"publisher","unstructured":"Nguyen V, Huber S, Gambi A (2021) SALVO: automated generation of diversified tests for self-driving cars from existing maps. In: International conference on artificial intelligence testing. IEEE, pp 128\u2013135. https:\/\/doi.org\/10.1109\/AITEST52744.2021.00033","DOI":"10.1109\/AITEST52744.2021.00033"},{"key":"10697_CR61","doi-asserted-by":"publisher","unstructured":"Piazzoni A, Cherian J, Azhar M, Yap JY, Shung JLW, Vijay R (2021) Vista: a framework for virtual scenario-based testing of autonomous vehicles. In: International conference on artificial intelligence testing. IEEE, pp 143\u2013150. https:\/\/doi.org\/10.1109\/AITEST52744.2021.00035","DOI":"10.1109\/AITEST52744.2021.00035"},{"key":"10697_CR62","unstructured":"PX4 (2021) Px4 simulation. https:\/\/docs.px4.io\/v1.12\/en\/simulation\/"},{"key":"10697_CR63","unstructured":"PX4.io (2022) Log analysis using flight review | px4 user guide. https:\/\/docs.px4.io\/master\/en\/log\/flight_review.html. Accessed: 07-Feb-2022"},{"key":"10697_CR64","unstructured":"PX4.io (2022) Obstacle avoidance | px4 user guide. https:\/\/docs.px4.io\/master\/en\/computer_vision\/obstacle_avoidance.html. Accessed: 07-Feb-2022"},{"key":"10697_CR65","doi-asserted-by":"publisher","unstructured":"Roy D, Hobbs C, Anderson JH, Caccamo M, Chakraborty S (2021) Timing debugging for cyber-physical systems. In: Design, automation & test in europe conference & exhibition, DATE 2021, Grenoble, France, February 1-5, 2021. IEEE, pp 1893\u20131898. https:\/\/doi.org\/10.23919\/DATE51398.2021.9474012","DOI":"10.23919\/DATE51398.2021.9474012"},{"key":"10697_CR66","doi-asserted-by":"crossref","unstructured":"Ryan P, Shahbeigi S, Zou J, Stefanakos I, Molloy J (2024) A dynamic assurance framework for an autonomous survey drone. In: International conference on computer safety, reliability, and security. Springer, pp 285\u2013299","DOI":"10.1007\/978-3-031-68606-1_18"},{"key":"10697_CR67","unstructured":"SAE International (1996) ARP4761: guidelines and methods for conducting the safety assessment process on civil airborne systems and equipment. SAE International, Warrendale, PA. Aerospace Recommended Practice"},{"key":"10697_CR68","unstructured":"SAE International (2010) ARP4754A: guidelines for development of civil aircraft and systems. SAE International, Warrendale, PA. Aerospace Recommended Practice"},{"key":"10697_CR69","doi-asserted-by":"crossref","unstructured":"Shar LK, Minn W, Ta NBD, Fan J, Jiang L, Kiat DLW (2022) Dronlomaly: runtime detection of anomalous drone behaviors via log analysis and deep learning. In: 2022 29th Asia-Pacific software engineering conference (APSEC). IEEE, pp 119\u2013128","DOI":"10.1109\/APSEC57359.2022.00024"},{"key":"10697_CR70","doi-asserted-by":"publisher","first-page":"110813","DOI":"10.1016\/j.jss.2020.110813","volume":"171","author":"SY Shin","year":"2021","unstructured":"Shin SY, Chaouch K, Nejati S, Sabetzadeh M, Briand LC, Zimmer F (2021) Uncertainty-aware specification and analysis for hardware-in-the-loop testing of cyber-physical systems. J Syst Softw 171:110813. https:\/\/doi.org\/10.1016\/j.jss.2020.110813","journal-title":"J Syst Softw"},{"key":"10697_CR71","doi-asserted-by":"crossref","unstructured":"Sindhwani V, Sidahmed H, Choromanski K, Jones B (2020) Unsupervised anomaly detection for self-flying delivery drones. In: 2020 IEEE international conference on robotics and automation (ICRA). IEEE, pp 186\u2013192","DOI":"10.1109\/ICRA40945.2020.9197074"},{"key":"10697_CR72","doi-asserted-by":"publisher","unstructured":"Smith SC, Ramamoorthy S (2021) Attainment regions in feature-parameter space for high-level debugging in autonomous robots. In: IEEE\/RSJ international conference on intelligent robots and systems, IROS 2021, Prague, Czech Republic, September 27 - Oct. 1, 2021. IEEE, pp 6546\u20136551. https:\/\/doi.org\/10.1109\/IROS51168.2021.9636336","DOI":"10.1109\/IROS51168.2021.9636336"},{"key":"10697_CR73","doi-asserted-by":"crossref","unstructured":"Stocco A, Weiss M, Calzana M, Tonella P (2020) Misbehaviour prediction for autonomous driving systems. In: International conference on software engineering, pp 359\u2013371","DOI":"10.1145\/3377811.3380353"},{"key":"10697_CR74","doi-asserted-by":"publisher","unstructured":"Su S, Li Y, He S, Han S, Feng C, Ding C, Miao F (2023) Uncertainty quantification of collaborative detection for self-driving. In: IEEE International Conference on Robotics and Automation, ICRA 2023, London, UK, May 29 - June 2, 2023. IEEE, pp 5588\u20135594. https:\/\/doi.org\/10.1109\/ICRA48891.2023.10160367","DOI":"10.1109\/ICRA48891.2023.10160367"},{"key":"10697_CR75","doi-asserted-by":"publisher","first-page":"107751","DOI":"10.1016\/j.asoc.2021.107751","volume":"112","author":"M Thill","year":"2021","unstructured":"Thill M, Konen W, Wang H, B\u00e4ck T (2021) Temporal convolutional autoencoder for unsupervised anomaly detection in time series. Appl Soft Comput 112:107751","journal-title":"Appl Soft Comput"},{"key":"10697_CR76","doi-asserted-by":"crossref","unstructured":"Timperley CS, Afzal A, Katz DS, Hernandez JM, Le\u00a0Goues C (2018) Crashing simulated planes is cheap: Can simulation detect robotics bugs early? In: 2018 IEEE 11th International conference on software testing, verification and validation (ICST). IEEE, pp 331\u2013342","DOI":"10.1109\/ICST.2018.00040"},{"key":"10697_CR77","doi-asserted-by":"crossref","unstructured":"Titouna C, Na\u00eft-Abdesselam F, Moungla H (2020) An online anomaly detection approach for unmanned aerial vehicles. In: 2020 International wireless communications and mobile computing (IWCMC). IEEE, pp 469\u2013474","DOI":"10.1109\/IWCMC48107.2020.9148073"},{"key":"10697_CR78","doi-asserted-by":"publisher","unstructured":"Tlili F, Ayed S, Chaari L, Ouni B (2022) Artificial intelligence based approach for fault and anomaly detection within uavs. In: Barolli L, Hussain F, Enokido T (eds) Advanced information networking and applications - proceedings of the 36th international conference on advanced information networking and applications (AINA-2022), Sydney, NSW, Australia, 13-15 April 2022, Volume 1, Lecture Notes in Networks and Systems. Springer, vol 449, pp 297\u2013308. https:\/\/doi.org\/10.1007\/978-3-030-99584-3_26","DOI":"10.1007\/978-3-030-99584-3_26"},{"key":"10697_CR79","unstructured":"UK CCDW (2019) Accidents will happen - a review of military drone crash data as the uk considers allowing large military drone flights in its airspace. https:\/\/dronewars.net\/wp-content\/uploads\/2019\/06\/DW-Accidents-WEB.pdf"},{"key":"10697_CR80","doi-asserted-by":"crossref","unstructured":"Wang D, Li S, Xiao G, Liu Y, Sui Y (2021) An exploratory study of autopilot software bugs in unmanned aerial vehicles. In: ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering, pp 20\u201331","DOI":"10.1145\/3468264.3468559"},{"key":"10697_CR81","doi-asserted-by":"crossref","unstructured":"Weiss M, Tonella P (2021) Fail-safe execution of deep learning based systems through uncertainty monitoring. In: 2021 14th IEEE conference on software testing, verification and validation (ICST). IEEE, pp 24\u201335","DOI":"10.1109\/ICST49551.2021.00015"},{"key":"10697_CR82","doi-asserted-by":"publisher","unstructured":"Weiss M, Tonella P (2021) Uncertainty-wizard: Fast and user-friendly neural network uncertainty quantification. In: 14th IEEE conference on software testing, verification and validation, ICST 2021, Porto de Galinhas, Brazil, April 12-16, 2021, pp. 436\u2013441. IEEE. https:\/\/doi.org\/10.1109\/ICST49551.2021.00056","DOI":"10.1109\/ICST49551.2021.00056"},{"key":"10697_CR83","doi-asserted-by":"publisher","unstructured":"Wilson EB (1927) Probable inference, the law of succession, and statistical inference. J Am Stat Assoc 22(158):209\u2013212. https:\/\/doi.org\/10.1080\/01621459.1927.10502953https:\/\/www.tandfonline.com\/doi\/abs\/10.1080\/01621459.1927.10502953","DOI":"10.1080\/01621459.1927.10502953"},{"key":"10697_CR84","doi-asserted-by":"crossref","unstructured":"Woodlief T, Elbaum S, Sullivan K (2021) Fuzzing mobile robot environments for fast automated crash detection. In: International conference on robotics and automation. IEEE, pp 5417\u20135423","DOI":"10.1109\/ICRA48506.2021.9561627"},{"key":"10697_CR85","unstructured":"Wotawa F (2021) On the use of available testing methods for verification & validation of ai-based software and systems. In: Workshop on artificial intelligence safety, CEUR Workshop Proceedings, vol. 2808. CEUR-WS.org. http:\/\/ceur-ws.org\/Vol-2808\/Paper_29.pdf"},{"key":"10697_CR86","doi-asserted-by":"publisher","first-page":"111769","DOI":"10.1016\/J.JSS.2023.111769","volume":"204","author":"L Wu","year":"2023","unstructured":"Wu L, Xi Z, Zheng Z, Li X (2023) Application of metamorphic testing on UAV path planning software. J Syst Softw 204:111769. https:\/\/doi.org\/10.1016\/J.JSS.2023.111769","journal-title":"J Syst Softw"},{"key":"10697_CR87","doi-asserted-by":"publisher","unstructured":"Xu Q, Ali S, Yue T, Arratibel M (2022) Uncertainty-aware transfer learning to evolve digital twins for industrial elevators. In: Roychoudhury A, Cadar C, Kim M (eds) Proceedings of the 30th ACM joint european software engineering conference and symposium on the foundations of software engineering, ESEC\/FSE 2022, Singapore, Singapore, November 14-18, 2022. ACM, pp 1257\u20131268. https:\/\/doi.org\/10.1145\/3540250.3558957","DOI":"10.1145\/3540250.3558957"},{"key":"10697_CR88","doi-asserted-by":"publisher","first-page":"120668","DOI":"10.1016\/j.eswa.2023.120668","volume":"230","author":"T Yang","year":"2023","unstructured":"Yang T, Hao W, Yang Q, Wang W (2023) Cloud-edge coordinated traffic anomaly detection for industrial cyber-physical systems. Expert Syst Appl 230:120668. https:\/\/doi.org\/10.1016\/j.eswa.2023.120668","journal-title":"Expert Syst Appl"},{"key":"10697_CR89","doi-asserted-by":"publisher","unstructured":"Zampetti F, Kapur R, Di Penta M, Panichella S (2022) An empirical characterization of software bugs in open-source cyber\u2013physical systems. J Syst Softw 192. https:\/\/doi.org\/10.1016\/j.jss.2022.111425","DOI":"10.1016\/j.jss.2022.111425"},{"key":"10697_CR90","doi-asserted-by":"publisher","unstructured":"Zampetti F, Tamburri DA, Panichella S, Panichella A, Canfora G, Penta MD (2022) Continuous integration and delivery practices for cyber-physical systems: an interview-based study. ACM Trans Softw Eng Methodol. https:\/\/doi.org\/10.1145\/3571854","DOI":"10.1145\/3571854"},{"issue":"2","key":"10697_CR91","doi-asserted-by":"publisher","first-page":"49:1","DOI":"10.1145\/3544792","volume":"32","author":"T Zohdinasab","year":"2023","unstructured":"Zohdinasab T, Riccio V, Gambi A, Tonella P (2023) Efficient and effective feature space exploration for testing deep learning systems. ACM Trans Softw Eng Methodol 32(2):49:1-49:38. https:\/\/doi.org\/10.1145\/3544792","journal-title":"ACM Trans Softw Eng Methodol"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-025-10697-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-025-10697-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-025-10697-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,5]],"date-time":"2025-11-05T02:41:47Z","timestamp":1762310507000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-025-10697-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,25]]},"references-count":93,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2025,12]]}},"alternative-id":["10697"],"URL":"https:\/\/doi.org\/10.1007\/s10664-025-10697-z","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,25]]},"assertion":[{"value":"1 July 2025","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 September 2025","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declared that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflicts of interest"}}],"article-number":"166"}}