{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,17]],"date-time":"2026-07-17T15:22:19Z","timestamp":1784301739294,"version":"3.55.0"},"reference-count":46,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2025,5,24]],"date-time":"2025-05-24T00:00:00Z","timestamp":1748044800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2025,6,30]]},"abstract":"<jats:p>As the era of autonomous cyber-physical systems (ACPSs), such as unmanned aerial vehicles and self-driving cars, unfolds, the demand for robust testing methodologies is key to realizing the adoption of such systems in real-world scenarios. However, traditional software testing paradigms face unprecedented challenges in ensuring the safety and reliability of these systems. In response, this article pioneers a strategic roadmap for simulation-based system-level testing of ACPSs, specifically focusing on autonomous systems. Our article discusses the relevant challenges and obstacles of ACPSs, focusing on test automation and quality assurance, hence advocating for tailored solutions to address the unique demands of autonomous systems. While providing concrete definitions of test cases within simulation environments, we also accentuate the need to create new benchmark assets and the development of automated tools tailored explicitly for autonomous systems in the software engineering community. This article not only highlights the relevant, pressing issues the software engineering community should focus on (in terms of practices, expected automation, and paradigms), but it also outlines ways to tackle them. By outlining the various domains and challenges of simulation-based testing\/development for ACPSs, we provide directions for future research efforts.<\/jats:p>","DOI":"10.1145\/3711906","type":"journal-article","created":{"date-parts":[[2025,1,16]],"date-time":"2025-01-16T14:48:33Z","timestamp":1737038913000},"page":"1-9","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":27,"title":["A Roadmap for Simulation-Based Testing of Autonomous Cyber-Physical Systems: Challenges and Future Direction"],"prefix":"10.1145","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3987-0276","authenticated-orcid":false,"given":"Christian","family":"Birchler","sequence":"first","affiliation":[{"name":"Zurich University of Applied Sciences, Winterthur, Switzerland and University of Bern, Bern, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0354-9747","authenticated-orcid":false,"given":"Sajad","family":"Khatiri","sequence":"additional","affiliation":[{"name":"Zurich University of Applied Sciences, Winterthur, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5127-4042","authenticated-orcid":false,"given":"Pooja","family":"Rani","sequence":"additional","affiliation":[{"name":"University of Zurich, Zurich, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2582-5557","authenticated-orcid":false,"given":"Timo","family":"Kehrer","sequence":"additional","affiliation":[{"name":"University of Bern, Bern, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4120-626X","authenticated-orcid":false,"given":"Sebastiano","family":"Panichella","sequence":"additional","affiliation":[{"name":"University of Bern, Bern, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,5,24]]},"reference":[{"key":"e_1_3_1_2_2","first-page":"263","volume-title":"Conference on Software Testing, Verification and Validation","author":"Afzal Afsoon","year":"2021","unstructured":"Afsoon Afzal, Deborah S. Katz, Claire Le Goues, and Christopher S. Timperley. 2021. Simulation for robotics test automation: Developer perspectives. In Conference on Software Testing, Verification and Validation. IEEE, 263\u2013274."},{"key":"e_1_3_1_3_2","first-page":"96","volume-title":"International Conference on Software Testing, Validation and Verification","author":"Afzal Afsoon","year":"2020","unstructured":"Afsoon Afzal, Claire Le Goues, Michael Hilton, and Christopher Steven Timperley. 2020. A study on challenges of testing robotic systems. In International Conference on Software Testing, Validation and Verification. IEEE, 96\u2013107."},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.3390\/challe6010117"},{"key":"e_1_3_1_5_2","first-page":"284","volume-title":"International Conference on Software Analysis, Evolution and Reengineering","author":"Arrieta Aitor","year":"2022","unstructured":"Aitor Arrieta, Maialen Otaegi, Liping Han, Goiuria Sagardui, Shaukat Ali, and Maite Arratibel. 2022. Automating test Oracle generation in DevOps for industrial elevators. In International Conference on Software Analysis, Evolution and Reengineering. IEEE, 284\u2013288."},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/2934466.2946046"},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2014.2372785"},{"key":"e_1_3_1_8_2","first-page":"45","volume-title":"International Workshop on Search-Based and Fuzz Testing","author":"Biagiola Matteo","year":"2023","unstructured":"Matteo Biagiola, Stefan Klikovits, Jarkko Peltom\u00e4ki, and Vincenzo Riccio. 2023. SBFT tool competition 2023 - cyber-physical systems track. In International Workshop on Search-Based and Fuzz Testing. IEEE, 45\u201348."},{"issue":"3","key":"e_1_3_1_9_2","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1007\/s10664-023-10286-y","article-title":"Machine learning-based test selection for simulation-based testing of self-driving cars software","volume":"28","author":"Birchler Christian","year":"2023","unstructured":"Christian Birchler, Sajad Khatiri, Bill Bosshard, Alessio Gambi, and Sebastiano Panichella. 2023. Machine learning-based test selection for simulation-based testing of self-driving cars software. Empir. Softw. Eng. 28, 3 (2023), 71.","journal-title":"Empir. Softw. Eng"},{"issue":"2","key":"e_1_3_1_10_2","first-page":"28:1","article-title":"Single and multi-objective test cases prioritization for self-driving cars in virtual environments","volume":"32","author":"Birchler Christian","year":"2023","unstructured":"Christian Birchler, Sajad Khatiri, Pouria Derakhshanfar, Sebastiano Panichella, and Annibale Panichella. 2023. Single and multi-objective test cases prioritization for self-driving cars in virtual environments. ACM Trans. Softw. Eng. Methodol. 32, 2 (2023), 28:1\u201328:30.","journal-title":"ACM Trans. Softw. Eng. Methodol"},{"key":"e_1_3_1_11_2","first-page":"929","volume-title":"International Conference on the Foundations of Software Engineering","author":"Birchler Christian","year":"2024","unstructured":"Christian Birchler, Tanzil Kombarabettu Mohammed, Pooja Rani, Teodora Nechita, Timo Kehrer, and Sebastiano Panichella. 2024. How does simulation-based testing for Self-driving cars match human perception?. In International Conference on the Foundations of Software Engineering, 929\u2013950."},{"key":"e_1_3_1_12_2","doi-asserted-by":"crossref","first-page":"510","DOI":"10.1145\/3643991.3644891","volume-title":"International Conference on Mining Software Repositories","author":"Birchler Christian","year":"2024","unstructured":"Christian Birchler, Cyrill Rohrbach, Timo Kehrer, and Sebastiano Panichella. 2024. SensoDat: Simulation-based sensor dataset of self-driving cars. In International Conference on Mining Software Repositories, 510\u2013514."},{"key":"e_1_3_1_13_2","first-page":"4243","volume-title":"International Conference on Robotics and Automation","author":"Bousmalis Konstantinos","year":"2018","unstructured":"Konstantinos Bousmalis, Alex Irpan, Paul Wohlhart, Yunfei Bai, Matthew Kelcey, Mrinal Kalakrishnan, Laura Downs, Julian Ibarz, Peter Pastor, Kurt Konolige, et al. 2018. Using simulation and domain adaptation to improve efficiency of deep robotic grasping. In International Conference on Robotics and Automation. IEEE, 4243\u20134250."},{"key":"e_1_3_1_14_2","unstructured":"Jack Collins Ross Brown Jurgen Leitner and David Howard. 2020. Traversing the reality gap via simulator tuning. arXiv:2003.01369. Retrieved from https:\/\/arxiv.org\/abs\/2003.01369"},{"key":"e_1_3_1_15_2","first-page":"133","volume-title":"International Conference on Robotics in Alpe-Adria Danube Region","author":"Dimitropoulos Konstantinos","year":"2022","unstructured":"Konstantinos Dimitropoulos, Ioannis Hatzilygeroudis, and Konstantinos Chatzilygeroudis. 2022. A brief survey of Sim2Real methods for robot learning. In International Conference on Robotics in Alpe-Adria Danube Region. Springer, 133\u2013140."},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3343753"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2782813"},{"key":"e_1_3_1_18_2","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1145\/3526072.3527538","volume-title":"International Workshop on Search-Based Software Testing","author":"Gambi Alessio","year":"2022","unstructured":"Alessio Gambi, Gunel Jahangirova, Vincenzo Riccio, and Fiorella Zampetti. 2022. SBST tool competition 2022. In International Workshop on Search-Based Software Testing. IEEE, 25\u201332."},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3337773"},{"issue":"3","key":"e_1_3_1_20_2","first-page":"33","article-title":"Co-simulation: A survey","volume":"51","author":"Gomes Cl\u00e1udio","year":"2018","unstructured":"Cl\u00e1udio Gomes, Casper Thule, David Broman, Peter Gorm Larsen, and Hans Vangheluwe. 2018. Co-simulation: A survey. ACM Comput. Surv. 51, 3, Article 49 (May 2018), 33 pages.","journal-title":"ACM Comput. Surv"},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2946773"},{"key":"e_1_3_1_22_2","first-page":"10912","volume-title":"International Conference on Robotics and Automation","author":"Hildebrandt Carl","year":"2021","unstructured":"Carl Hildebrandt and Sebastian Elbaum. 2021. World-in-the-loop simulation for autonomous systems validation. In International Conference on Robotics and Automation. IEEE, 10912\u201310919."},{"issue":"2","key":"e_1_3_1_23_2","doi-asserted-by":"crossref","first-page":"398","DOI":"10.1109\/TASE.2021.3064065","article-title":"Sim2Real in robotics and automation: Applications and challenges","volume":"18","author":"H\u00f6fer Sebastian","year":"2021","unstructured":"Sebastian H\u00f6fer, Kostas Bekris, Ankur Handa, Juan Camilo Gamboa, Melissa Mozifian, Florian Golemo, Chris Atkeson, Dieter Fox, Ken Goldberg, John Leonard, et al. 2021. Sim2Real in robotics and automation: Applications and challenges. IEEE Trans. Autom. Sci. Eng. 18, 2 (2021), 398\u2013400.","journal-title":"IEEE Trans. Autom. Sci. Eng"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1982.235571"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/2931037.2931062"},{"key":"e_1_3_1_26_2","first-page":"194","volume-title":"Conference on Software Testing, Verification and Validation","author":"Jahangirova Gunel","year":"2021","unstructured":"Gunel Jahangirova, Andrea Stocco, and Paolo Tonella. 2021. Quality metrics and oracles for autonomous vehicles testing. In Conference on Software Testing, Verification and Validation. IEEE, 194\u2013204."},{"key":"e_1_3_1_27_2","first-page":"334","volume-title":"Conference on Robot Learning","author":"James Stephen","year":"2017","unstructured":"Stephen James, Andrew J. Davison, and Edward Johns. 2017. Transferring end-to-end visuomotor control from simulation to real world for a multi-stage task. In Conference on Robot Learning. PMLR, 334\u2013343."},{"key":"e_1_3_1_28_2","first-page":"281","volume-title":"IEEE Conference on Software Testing, Verification and Validation","author":"Khatiri Sajad","year":"2023","unstructured":"Sajad Khatiri, Sebastiano Panichella, and Paolo Tonella. 2023. Simulation-based test case generation for unmanned aerial vehicles in the neighborhood of real flights. In IEEE Conference on Software Testing, Verification and Validation. IEEE, 281\u2013292."},{"key":"e_1_3_1_29_2","volume-title":"International Conference on Software Engineering","author":"Khatiri Sajad","year":"2024","unstructured":"Sajad Khatiri, Sebastiano Panichella, and Paolo Tonella. 2024. Simulation-based testing of unmanned aerial vehicles with aerialist. In International Conference on Software Engineering."},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3643659.3643931"},{"key":"e_1_3_1_31_2","first-page":"9426","volume-title":"International Conference on Robotics and Automation","author":"Lee Timothy E.","year":"2020","unstructured":"Timothy E. Lee, Jonathan Tremblay, Thang To, Jia Cheng, Terry Mosier, Oliver Kroemer, Dieter Fox, and Stan Birchfield. 2020. Camera-to-robot pose estimation from a single image. In International Conference on Robotics and Automation. IEEE, 9426\u20139432."},{"key":"e_1_3_1_32_2","first-page":"52","volume-title":"International Conference on Mining Software Repositories","author":"Lu Chengjie","year":"2023","unstructured":"Chengjie Lu, Tao Yue, and Shaukat Ali. 2023. DeepScenario: An open driving scenario dataset for autonomous driving system testing. In International Conference on Mining Software Repositories. IEEE, 52\u201356."},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3298432"},{"key":"e_1_3_1_34_2","first-page":"237","volume-title":"International Conference on Software Engineering","author":"Manotas Irene","year":"2016","unstructured":"Irene Manotas, Christian Bird, Rui Zhang, David Shepherd, Ciera Jaspan, Caitlin Sadowski, Lori Pollock, and James Clause. 2016. An empirical study of practitioners\u2019 perspectives on Green software engineering. In International Conference on Software Engineering, 237\u2013248."},{"key":"e_1_3_1_35_2","first-page":"4008","volume-title":"International Intelligent Transportation Systems Conference","author":"Ngo Anthony","year":"2021","unstructured":"Anthony Ngo, Max Paul Bauer, and Michael Resch. 2021. A multi-layered approach for measuring the simulation-to-reality gap of radar perception for autonomous driving. In International Intelligent Transportation Systems Conference. IEEE, 4008\u20134014."},{"key":"e_1_3_1_36_2","first-page":"20","volume-title":"International Workshop on Search-Based Software Testing","author":"Panichella Sebastiano","year":"2021","unstructured":"Sebastiano Panichella, Alessio Gambi, Fiorella Zampetti, and Vincenzo Riccio. 2021. SBST tool competition 2021. In International Workshop on Search-Based Software Testing. IEEE, 20\u201327."},{"key":"e_1_3_1_37_2","first-page":"5537","volume-title":"IEEE\/RSJ International Conference on Intelligent Robots and Systems","author":"Parra Samuel","year":"2023","unstructured":"Samuel Parra, Argentina Ortega, Sven Schneider, and Nico Hochgeschwender. 2023. A thousand worlds: Scenery specification and generation for simulation-based testing of mobile robot navigation stacks. In IEEE\/RSJ International Conference on Intelligent Robots and Systems, 5537\u20135544."},{"key":"e_1_3_1_38_2","doi-asserted-by":"crossref","unstructured":"Pooja Rani Jonas Zellweger Veronika Kousadianos Luis Cruz Timo Kehrer and Alberto Bacchelli. 2024. Energy patterns for web: An exploratory study. arXiv:2401.06482. Retrieved from https:\/\/arxiv.org\/abs\/2401.06482","DOI":"10.1145\/3639475.3640110"},{"key":"e_1_3_1_39_2","first-page":"1249","volume-title":"IEEE Intelligent Vehicles Symposium","author":"Reway Fabio","year":"2020","unstructured":"Fabio Reway, Abdul Hoffmann, Diogo Wachtel, Werner Huber, Alois C. Knoll, and Eduardo Parente Ribeiro. 2020. Test method for measuring the simulation-to-reality gap of camera-based object detection algorithms for autonomous driving. In IEEE Intelligent Vehicles Symposium. IEEE, 1249\u20131256."},{"key":"e_1_3_1_40_2","doi-asserted-by":"crossref","first-page":"153171","DOI":"10.1109\/ACCESS.2021.3126658","article-title":"Crossing the reality gap: A survey on sim-to-real transferability of robot controllers in reinforcement learning","author":"Salvato Erica","year":"2021","unstructured":"Erica Salvato, Gianfranco Fenu, Eric Medvet, and Felice Andrea Pellegrino. 2021. Crossing the reality gap: A survey on sim-to-real transferability of robot controllers in reinforcement learning. IEEE Access 9 (2021), 153171\u2013153187.","journal-title":"IEEE Access 9"},{"issue":"3","key":"e_1_3_1_41_2","first-page":"67:1","article-title":"Automated identification and qualitative characterization of safety concerns reported in UAV software platforms","volume":"32","author":"Sorbo Andrea Di","year":"2023","unstructured":"Andrea Di Sorbo, Fiorella Zampetti, Aaron Visaggio, Massimiliano Di Penta, and Sebastiano Panichella. 2023. Automated identification and qualitative characterization of safety concerns reported in UAV software platforms. ACM Trans. Softw. Eng. Methodol. 32, 3 (2023), 67:1\u201367:37.","journal-title":"ACM Trans. Softw. Eng. Methodol"},{"key":"e_1_3_1_42_2","first-page":"20","volume-title":"ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering","author":"Wang Dinghua","year":"2021","unstructured":"Dinghua Wang, Shuqing Li, Guanping Xiao, Yepang Liu, and Yulei Sui. 2021. An exploratory study of autopilot software bugs in unmanned aerial vehicles. In ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 20\u201331."},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","DOI":"10.1002\/stv.430"},{"key":"e_1_3_1_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3644032.3644461"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2022.111425"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.1177\/0278364919870227"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2019.03.011"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3711906","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3711906","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:18:09Z","timestamp":1750295889000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3711906"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,24]]},"references-count":46,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2025,6,30]]}},"alternative-id":["10.1145\/3711906"],"URL":"https:\/\/doi.org\/10.1145\/3711906","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5,24]]},"assertion":[{"value":"2024-10-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-11-20","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-24","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}