{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,28]],"date-time":"2026-03-28T06:35:21Z","timestamp":1774679721756,"version":"3.50.1"},"reference-count":79,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2024,7,12]],"date-time":"2024-07-12T00:00:00Z","timestamp":1720742400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100010661","name":"Horizon 2020 Framework Programme","doi-asserted-by":"publisher","award":["957254-COSMOS"],"award-info":[{"award-number":["957254-COSMOS"]}],"id":[{"id":"10.13039\/100010661","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2024,7,12]]},"abstract":"<jats:p>\n                    Software metrics such as coverage or mutation scores have been investigated for the automated quality assessment of test suites. While traditional tools rely on software metrics, the field of self-driving cars (SDCs) has primarily focused on simulation-based test case generation using quality metrics such as the out-of-bound (OOB) parameter to determine if a test case fails or passes. However, it remains unclear to what extent this quality metric aligns with the human perception of the safety and realism of SDCs. To address this (reality) gap, we conducted an empirical study involving 50 participants to investigate the factors that determine how humans perceive SDC test cases as safe, unsafe, realistic, or unrealistic. To this aim, we developed a framework leveraging virtual reality (VR) technologies, called SDC-A\n                    <jats:sc>labaster<\/jats:sc>\n                    , to immerse the study participants into the virtual environment of SDC simulators. Our findings indicate that the human assessment of safety and realism of failing\/passing test cases can vary based on different factors, such as the test\u2019s complexity and the possibility of interacting with the SDC. Especially for the assessment of realism, the participants' age leads to a different perception. This study highlights the need for more research on simulation testing quality metrics and the importance of human perception in evaluating SDCs.\n                  <\/jats:p>","DOI":"10.1145\/3643768","type":"journal-article","created":{"date-parts":[[2024,7,12]],"date-time":"2024-07-12T10:22:09Z","timestamp":1720779729000},"page":"929-950","source":"Crossref","is-referenced-by-count":20,"title":["How Does Simulation-Based Testing for Self-Driving Cars Match Human Perception?"],"prefix":"10.1145","volume":"1","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3987-0276","authenticated-orcid":false,"given":"Christian","family":"Birchler","sequence":"first","affiliation":[{"name":"Zurich University of Applied Sciences, Winterthur, Switzerland"},{"name":"University of Bern, Bern, Switzerland"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-1536-4327","authenticated-orcid":false,"given":"Tanzil Kombarabettu","family":"Mohammed","sequence":"additional","affiliation":[{"name":"University of Zurich, Zurich, Switzerland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5127-4042","authenticated-orcid":false,"given":"Pooja","family":"Rani","sequence":"additional","affiliation":[{"name":"University of Zurich, Zurich, Switzerland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2526-9308","authenticated-orcid":false,"given":"Teodora","family":"Nechita","sequence":"additional","affiliation":[{"name":"Zurich University of Applied Sciences, Winterthur, Switzerland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2582-5557","authenticated-orcid":false,"given":"Timo","family":"Kehrer","sequence":"additional","affiliation":[{"name":"University of Bern, Bern, Switzerland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4120-626X","authenticated-orcid":false,"given":"Sebastiano","family":"Panichella","sequence":"additional","affiliation":[{"name":"Zurich University of Applied Sciences, Winterthur, Switzerland"}]}],"member":"320","published-online":{"date-parts":[[2024,7,12]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","unstructured":"Raja Ben Abdessalem Shiva Nejati Lionel C. Briand and Thomas Stifter. 2018. Testing vision-based control systems using learnable evolutionary algorithms. In International Conference on Software Engineering. 1016-1026. https:\/\/doi.org\/10.1145\/3180155.3180160 10.1145\/3180155.3180160","DOI":"10.1145\/3180155.3180160"},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397386"},{"key":"e_1_3_1_4_2","unstructured":"Afsoon Afzal Deborah S. Katz Claire Le Goues and Christopher Steven Timperley. 2020. A Study on the Challenges of Using Robotics Simulators for Testing. arXiv:2004.07368 https:\/\/arxiv.org\/abs\/2004.07368"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST49551.2021.00036"},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/DSD53832.2021.00071"},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER53432.2022.00044"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2018.09.055"},{"key":"e_1_3_1_9_2","unstructured":"BBC. 2023. Robots to do 39% of domestic chores by 2033 say experts. https:\/\/www.bbc.com\/news\/technology-64718842. Accessed: 2023-01-04."},{"key":"e_1_3_1_10_2","unstructured":"BeamNG.tech. [n. d.]. BeamNG.research. https:\/\/documentation.beamng.com\/beamng_tech\/. Accessed: 2022-07-31."},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-023-10286-y"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533818"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","unstructured":"Christian Birchler Tanzil Kombarabettu Mohammed Pooja Rani Teodora Nechita Timo Kehrer and Sebastiano Panichella 2024 Replication Package - \u201cHow does Simulation-based Testing for Self-driving Cars match Human Perception?\u201d. https:\/\/doi.org\/10.5281\/zenodo.10570961 10.5281\/zenodo.10570961.","DOI":"10.5281\/zenodo.10570961"},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","unstructured":"Christian Birchler Tanzil Kombarabettu Mohammed Pooja Rani Teodora Nechita Timo Kehrer and Sebastiano Panichella 2024 Replication Package - \u201cHow does Simulation-based Testing for Self-driving Cars match Human Perception?\u201d. https:\/\/doi.org\/10.5281\/zenodo.10570960 10.5281\/zenodo.10570960.","DOI":"10.5281\/zenodo.10570960"},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","unstructured":"Christian Birchler Cyrill Rohrbach Hyeongkyun Kim Alessio Gambi Tianhai Liu Jens Horneber Timo Kehrer and Sebastiano Panichella. 2023. TEASER: Simulation-Based CAN Bus Regression Testing for Self-Driving Cars Software. In International Conference on Automated Software Engineering. 2058-2061. https:\/\/doi.org\/10.1109\/ASE56229.2023.00154 10.1109\/ASE56229.2023.00154","DOI":"10.1109\/ASE56229.2023.00154"},{"key":"e_1_3_1_16_2","first-page":"285","volume-title":"43. GIL-Jahrestagung, Resiliente Agri-Food-Systeme (LNI, Vol. P-330)","author":"Bohne Tim","year":"2023","unstructured":"Tim Bohne, Gurunatraj Parthasarathy, and Benjamin Kisliuk. 2023. A systematic approach to the development of long-term autonomous robotic systems for agriculture. In 43. GIL-Jahrestagung, Resiliente Agri-Food-Systeme (LNI, Vol. P-330). Gesellschaft f\u00fcr Informatik e.V., 285-290. https:\/\/dl.gi.de\/20.500.12116\/40260"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/SBST52555.2021.00016"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2019.8793789"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","unstructured":"Shafiul Azam Chowdhury Sohil Lal Shrestha Taylor T. Johnson and Christoph Csallner. 2020. SLEMI: equivalence modulo input (EMI) based mutation of CPS models for finding compiler bugs in Simulink. In International Conference on Software Engineering. 335-346. https:\/\/doi.org\/10.1145\/3377811.3380381 10.1145\/3377811.3380381","DOI":"10.1145\/3377811.3380381"},{"key":"e_1_3_1_20_2","unstructured":"Jack Collins Ross Brown Jurgen Leitner and David Howard. 2020. Traversing the reality gap via simulator tuning. arXiv preprint arXiv:2003.01369 (2020)."},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","unstructured":"Hugo Leonardo da Silva Araujo Mohammad Reza Mousavi and Mahsa Varshosaz. 2023. Testing Validation and Verification of Robotic and Autonomous Systems: A Systematic Review. ACM Trans. Softw. Eng. Methodol. 32 2 (2023) 51:1-51:61. https:\/\/doi.org\/10.1145\/3542945 10.1145\/3542945","DOI":"10.1145\/3542945"},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3126521"},{"key":"e_1_3_1_23_2","first-page":"1","volume-title":"Annual Conference on Robot Learning (Proceedings of Machine Learning Research, Vol. 78)","author":"Dosovitskiy Alexey","year":"2017","unstructured":"Alexey Dosovitskiy, Germ\u00e1n Ros, Felipe Codevilla, Antonio M. L\u00f3pez, and Vladlen Koltun. 2017. CARLA: An Open Urban Driving Simulator. In Annual Conference on Robot Learning (Proceedings of Machine Learning Research, Vol. 78). PMLR, 1-16. http:\/\/proceedings.mlr.press\/v78\/dosovitskiy17a.html"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-Companion.2019.00119"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338942"},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3526072.3527538"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380397"},{"key":"e_1_3_1_28_2","unstructured":"BeamNG GmbH. 2023. BeamNG.tech. https:\/\/beamng.tech\/"},{"key":"e_1_3_1_29_2","unstructured":"BeamNG GmbH. 2023. Publications based on BeamNG.tech. https:\/\/beamng.tech\/research\/"},{"key":"e_1_3_1_30_2","unstructured":"The Guardian. 2018. Self-driving Uber kills Arizona woman in first fatal crash involving pedestrian. https:\/\/www.theguardian.com\/technology\/2018\/mar\/19\/uber-self-driving-car-kills-woman-arizona-tempe."},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.3390\/s22218373"},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA48506.2021.9561240"},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3473128"},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3385956.3418945"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/VR.2019.8797996"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/2931037.2931062"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST49551.2021.00030"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST57152.2023.00034"},{"key":"e_1_3_1_39_2","doi-asserted-by":"crossref","unstructured":"Sajad Khatiri Sebastiano Panichella and Paolo Tonella. 2024. Simulation-based Testing of Unmanned Aerial Vehicles with Aerialist. In International Conference on Software Engineering (ICSE).","DOI":"10.1145\/3639478.3640031"},{"key":"e_1_3_1_40_2","doi-asserted-by":"crossref","unstructured":"Sajad Khatiri Prasun Saurabh Timothy Zimmermann Charith Munasinghe Christian Birchler and Sebastiano Panichella. 2024. SBFT Tool Competition 2024 - CPS-UAV Test Case Generation Track. In IEEE\/ACM International Workshop on Search-Based and Fuzz Testing SBFT@ICSE 2024.","DOI":"10.1145\/3643659.3643931"},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2012.2185849"},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA40945.2020.9196596"},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3387940.3392234"},{"key":"e_1_3_1_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3418907"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","unstructured":"Claudio Menghi Shiva Nejati Khouloud Gaaloul and Lionel C. Briand. 2019. Generating automated and online test oracles for Simulink models with continuous and uncertain behaviors. In Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 27-38. https:\/\/doi.org\/10.1145\/3338906.3338920 10.1145\/3338906.3338920","DOI":"10.1145\/3338906.3338920"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER48275.2020.9054812"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/AIVR50618.2020.00046"},{"key":"e_1_3_1_48_2","volume-title":"Workshop on Artificial Intelligence Safety (CEUR Workshop Proceedings, Vol. 2808)","author":"Nair Saasha","year":"2021","unstructured":"Saasha Nair, Sina Shafaei, Daniel Auge, and Alois C. Knoll. 2021. An Evaluation of \u201cCrash Prediction Networks\u201d (CPN) for Autonomous Driving Scenarios in CARLA Simulator. In Workshop on Artificial Intelligence Safety (CEUR Workshop Proceedings, Vol. 2808). CEUR-WS.org. http:\/\/ceur-ws.org\/Vol-2808\/Paper_10.pdf"},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/ITSC48978.2021.9564521"},{"key":"e_1_3_1_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/AITEST52744.2021.00033"},{"key":"e_1_3_1_51_2","unstructured":"Nvidia 2020. NVIDIA DRIVE Constellation. https:\/\/developer.nvidia.com\/drive\/drive-constellation"},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","unstructured":"Sebastiano Panichella Alessio Gambi Fiorella Zampetti and Vincenzo Riccio. 2021. SBST Tool Competition 2021. In International Workshop on Search-Based Software Testing. IEEE 20-27. https:\/\/doi.org\/10.1109\/SBST52555.2021.00011 10.1109\/SBST52555.2021.00011","DOI":"10.1109\/SBST52555.2021.00011"},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377813.3381346"},{"key":"e_1_3_1_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376847"},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/AITEST52744.2021.00035"},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/IV47402.2020.9304567"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","unstructured":"Guodong Rong Byung Hyun Shin Hadi Tabatabaee Qiang Lu Steve Lemke Martins Mozeiko Eric Boise Geehoon Uhm Mark Gerow Shalin Mehta Eugene Agafonov Tae Hyung Kim Eric Sterner Keunhae Ushiroda Michael Reyes Dmitry Zelenkovsky and Seonman Kim. 2020. LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving. (2020) 1-6. https:\/\/doi.org\/10.1109\/ITSC45102.2020.9294422 10.1109\/ITSC45102.2020.9294422","DOI":"10.1109\/ITSC45102.2020.9294422"},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","unstructured":"Erica Salvato Gianfranco Fenu Eric Medvet and Felice Andrea Pellegrino 2021 Crossing the Reality Gap: A Survey on Sim-to-Real Transferability of Robot Controllers in Reinforcement Learning. IEEE Access 9 (2021): 153171\u2013153187. https:\/\/doi.org\/10.1109\/ACCESS.2021.3126658 10.1109\/ACCESS.2021.3126658.","DOI":"10.1109\/ACCESS.2021.3126658"},{"key":"e_1_3_1_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3183399.3183414"},{"key":"e_1_3_1_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/CHASE.2019.00013"},{"key":"e_1_3_1_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/HRI53351.2022.9889526"},{"key":"e_1_3_1_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3564821"},{"key":"e_1_3_1_63_2","volume-title":"Card sorting: Designing usable categories","author":"Spencer Donna","year":"2009","unstructured":"Donna Spencer 2009 Card sorting: Designing usable categories. Rosenfeld Media."},{"key":"e_1_3_1_64_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10676-021-09602-1"},{"key":"e_1_3_1_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3202311"},{"key":"e_1_3_1_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380353"},{"key":"e_1_3_1_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3579642"},{"key":"e_1_3_1_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3449726.3462722"},{"key":"e_1_3_1_69_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409758"},{"key":"e_1_3_1_70_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-Companion52605.2021.00042"},{"key":"e_1_3_1_71_2","first-page":"331","volume-title":"International Conference on Software Testing, Verification and Validation","author":"Timperley Christopher Steven","year":"2018","unstructured":"Christopher Steven Timperley, Afsoon Afzal, Deborah S Katz, Jam Marcos Hernandez, and Claire Le Goues 2018 Crashing simulated planes is cheap: Can simulation detect robotics bugs early? In International Conference on Software Testing, Verification and Validation. IEEE, 331\u2013342."},{"key":"e_1_3_1_72_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468559"},{"key":"e_1_3_1_73_2","doi-asserted-by":"publisher","DOI":"10.1109\/MIM.2005.1438843"},{"key":"e_1_3_1_74_2","volume-title":"Workshop on Artificial Intelligence Safety (CEUR Workshop Proceedings, Vol. 2808)","author":"Wotawa Franz","year":"2021","unstructured":"Franz Wotawa. 2021. On the Use of Available Testing Methods for Verification & Validation of AI-based Software and Systems. In Workshop on Artificial Intelligence Safety (CEUR Workshop Proceedings, Vol. 2808). CEUR-WS.org. http:\/\/ceur-ws.org\/Vol-2808\/Paper_29.pdf"},{"key":"e_1_3_1_75_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST49551.2021.00031"},{"key":"e_1_3_1_76_2","doi-asserted-by":"publisher","unstructured":"Fiorella Zampetti Ritu Kapur Massimiliano Di Penta and Sebastiano Panichella 2022 An empirical characterization of software bugs in open-source Cyber-Physical Systems. Journal of Systems and Software 192 (2022) 111425. https:\/\/doi.org\/10.1016\/j.jss.2022.111425 10.1016\/j.jss.2022.111425.","DOI":"10.1016\/j.jss.2022.111425"},{"key":"e_1_3_1_77_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-60508-7_9"},{"key":"e_1_3_1_78_2","doi-asserted-by":"publisher","DOI":"10.1177\/0278364919870227"},{"key":"e_1_3_1_79_2","doi-asserted-by":"publisher","DOI":"10.1109\/INFOCOMWKSHPS50562.2020.9162743"},{"key":"e_1_3_1_80_2","doi-asserted-by":"publisher","unstructured":"Husheng Zhou Wei Li Zelun Kong Junfeng Guo Yuqun Zhang Bei Yu Lingming Zhang and Cong Liu. 2020. DeepBillboard: systematic physical-world testing of autonomous driving systems. In International Conference on Software Engineering. 347-358. https:\/\/doi.org\/10.1145\/3377811.3380422 10.1145\/3377811.3380422","DOI":"10.1145\/3377811.3380422"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3643768","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3643768","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,4]],"date-time":"2026-02-04T07:56:49Z","timestamp":1770191809000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3643768"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,12]]},"references-count":79,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2024,7,12]]}},"alternative-id":["10.1145\/3643768"],"URL":"https:\/\/doi.org\/10.1145\/3643768","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,12]]}}}