{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,10]],"date-time":"2026-11-10T17:43:14Z","timestamp":1794332594472,"version":"4.9.0"},"reference-count":238,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2023,3,30]],"date-time":"2023-03-30T00:00:00Z","timestamp":1680134400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"UKRI Trustworthy Autonomous Systems Node in Verifiability","award":["EP\/V026801\/2"],"award-info":[{"award-number":["EP\/V026801\/2"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2023,4,30]]},"abstract":"<jats:p>We perform a systematic literature review on testing, validation, and verification of robotic and autonomous systems (RAS). The scope of this review covers peer-reviewed research papers proposing, improving, or evaluating testing techniques, processes, or tools that address the system-level qualities of RAS.<\/jats:p>\n                  <jats:p>Our survey is performed based on a rigorous methodology structured in three phases. First, we made use of a set of 26 seed papers (selected by domain experts) and the SERP-TEST taxonomy to design our search query and (domain-specific) taxonomy. Second, we conducted a search in three academic search engines and applied our inclusion and exclusion criteria to the results. Respectively, we made use of related work and domain specialists (50 academics and 15 industry experts) to validate and refine the search query. As a result, we encountered 10,735 studies, out of which 195 were included, reviewed, and coded.<\/jats:p>\n                  <jats:p>Our objective is to answer four research questions, pertaining to (1) the type of models, (2) measures for system performance and testing adequacy, (3) tools and their availability, and (4) evidence of applicability, particularly in industrial contexts. We analyse the results of our coding to identify strengths and gaps in the domain and present recommendations to researchers and practitioners.<\/jats:p>\n                  <jats:p>Our findings show that variants of temporal logics are most widely used for modelling requirements and properties, while variants of state-machines and transition systems are used widely for modelling system behaviour. Other common models concern epistemic logics for specifying requirements and belief-desire-intention models for specifying system behaviour. Apart from time and epistemics, other aspects captured in models concern probabilities (e.g., for modelling uncertainty) and continuous trajectories (e.g., for modelling vehicle dynamics and kinematics).<\/jats:p>\n                  <jats:p>Many papers lack any rigorous measure of efficiency, effectiveness, or adequacy for their proposed techniques, processes, or tools. Among those that provide a measure of efficiency, effectiveness, or adequacy, the majority use domain-agnostic generic measures such as number of failures, size of state-space, or verification time were most used. There is a trend in addressing the research gap in this respect by developing domain-specific notions of performance and adequacy. Defining widely accepted rigorous measures of performance and adequacy for each domain is an identified research gap.<\/jats:p>\n                  <jats:p>In terms of tools, the most widely used tools are well-established model-checkers such as Prism and Uppaal, as well as simulation tools such as Gazebo; Matlab\/Simulink is another widely used toolset in this domain.<\/jats:p>\n                  <jats:p>Overall, there is very limited evidence of industrial applicability in the papers published in this domain. There is even a gap considering consolidated benchmarks for various types of autonomous systems.<\/jats:p>\n                  <jats:p\/>","DOI":"10.1145\/3542945","type":"journal-article","created":{"date-parts":[[2022,6,16]],"date-time":"2022-06-16T06:06:46Z","timestamp":1655359606000},"page":"1-61","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":97,"title":["Testing, Validation, and Verification of Robotic and Autonomous Systems: A Systematic Review"],"prefix":"10.1145","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3005-6008","authenticated-orcid":false,"given":"Hugo","family":"Araujo","sequence":"first","affiliation":[{"name":"King\u2019s College London"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4869-6794","authenticated-orcid":false,"given":"Mohammad Reza","family":"Mousavi","sequence":"additional","affiliation":[{"name":"King\u2019s College London"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4776-883X","authenticated-orcid":false,"given":"Mahsa","family":"Varshosaz","sequence":"additional","affiliation":[{"name":"IT University of Copenhagen"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,3,30]]},"reference":[{"issue":"2","key":"e_1_3_2_2_2","doi-asserted-by":"crossref","first-page":"197","DOI":"10.20485\/jsaeijae.10.2_197","article-title":"Verification of advanced driver assistance systems (ADAS) and autonomous vehicles with hardware emulation-in-the-loop","volume":"10","author":"AbdElSalam Mohamed","year":"2019","unstructured":"Mohamed AbdElSalam, Keroles Khalil, John Stickley, Ashraf Salem, and Bruno Loye. 2019. Verification of advanced driver assistance systems (ADAS) and autonomous vehicles with hardware emulation-in-the-loop. Int. J. Automot. Eng. 10, 2 (2019), 197\u2013204.","journal-title":"Int. J. Automot. Eng."},{"key":"e_1_3_2_3_2","first-page":"143","volume-title":"33rd IEEE\/ACM International Conference on Automated Software Engineering (ASE)","author":"Abdessalem Raja Ben","year":"2018","unstructured":"Raja Ben Abdessalem, Annibale Panichella, Shiva Nejati, Lionel C. Briand, and Thomas Stifter. 2018. Testing autonomous cars for feature interaction failures using many-objective search. In 33rd IEEE\/ACM International Conference on Automated Software Engineering (ASE). IEEE, 143\u2013154."},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-018-9670-1"},{"issue":"4","key":"e_1_3_2_5_2","first-page":"60617","article-title":"Autonomous vehicles scenario testing framework and model of computation","volume":"2","author":"Alnaser Ala Jamil","year":"2019","unstructured":"Ala Jamil Alnaser, Mustafa Ilhan Akbas, Arman Sargolzaei, and Rahul Razdan. 2019. Autonomous vehicles scenario testing framework and model of computation. SAE Int. J. Connect. Automat. Vehic. 2, 4 (2019), 60617\u201360628.","journal-title":"SAE Int. J. Connect. Automat. Vehic."},{"key":"e_1_3_2_6_2","first-page":"1162","volume-title":"14th International IEEE Conference on Intelligent Transportation Systems (ITSC)","author":"Althoff Matthias","year":"2011","unstructured":"Matthias Althoff and John M. Dolan. 2011. Set-based computation of vehicle behaviors for the online verification of autonomous vehicles. In 14th International IEEE Conference on Intelligent Transportation Systems (ITSC). IEEE, Washington, DC, 1162\u20131167."},{"issue":"31","key":"e_1_3_2_7_2","first-page":"56","article-title":"Effortless creation of safe robots from modules through self-programming and self-verification","volume":"4","author":"Althoff Matthias","year":"2019","unstructured":"Matthias Althoff, Andrea Giusti, Stefan B. Liu, and Aaron Pereira. 2019. Effortless creation of safe robots from modules through self-programming and self-verification. Sci. Robot. 4, 31 (2019), 56\u201389.","journal-title":"Sci. Robot."},{"key":"e_1_3_2_8_2","doi-asserted-by":"crossref","first-page":"392","DOI":"10.1109\/IBCAST.2017.7868084","volume-title":"14th International Bhurban Conference on Applied Sciences and Technology (IBCAST)","author":"Amin Saifullah","year":"2017","unstructured":"Saifullah Amin, Adnan Elahi, Kashif Saghar, and Faran Mehmood. 2017. Formal modelling and verification approach for improving probabilistic behaviour of robot swarms. In 14th International Bhurban Conference on Applied Sciences and Technology (IBCAST). IEEE, New York, NY, 392\u2013400."},{"key":"e_1_3_2_9_2","first-page":"185","volume-title":"International Conference on Principles and Practice of Multi-Agent Systems","author":"Aminof Benjamin","year":"2015","unstructured":"Benjamin Aminof, Aniello Murano, Sasha Rubin, and Florian Zuleger. 2015. Verification of asynchronous mobile-robots in partially-known environments. In International Conference on Principles and Practice of Multi-Agent Systems. Springer, 185\u2013200."},{"key":"e_1_3_2_10_2","first-page":"1190","volume-title":"International Conference on Autonomous Agents & Multiagent Systems","author":"Aminof Benjamin","year":"2016","unstructured":"Benjamin Aminof, Aniello Murano, Sasha Rubin, and Florian Zuleger. 2016. Automatic verification of multi-agent systems in parameterised grid-environments. In International Conference on Autonomous Agents & Multiagent Systems. ACM, 1190\u20131199."},{"key":"e_1_3_2_11_2","first-page":"663","volume-title":"4th International Conference on Model-Driven Engineering and Software Development (MODELSWARD)","author":"Andrews Anneliese","year":"2016","unstructured":"Anneliese Andrews, Mahmoud Abdelgawad, and Ahmed Gario. 2016. World model for testing urban search and rescue (USAR) robots using petri nets. In 4th International Conference on Model-Driven Engineering and Software Development (MODELSWARD). IEEE, 663\u2013670."},{"key":"e_1_3_2_12_2","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1109\/ICMSR.2019.8835477","volume-title":"IEEE 5th International Conference on Mechatronics System and Robots (ICMSR)","author":"Aparow Vimal Rau","year":"2019","unstructured":"Vimal Rau Aparow, Apratim Choudary, Giridharan Kulandaivelu, Thomas Webster, Justin Dauwels, and Niels de Boer. 2019. A comprehensive simulation platform for testing autonomous vehicles in 3D virtual environment. In IEEE 5th International Conference on Mechatronics System and Robots (ICMSR). IEEE, 115\u2013119."},{"key":"e_1_3_2_13_2","volume-title":"Concurrency, Specification and Programming Conference.","author":"Arai Ryota","year":"2017","unstructured":"Ryota Arai and H. Schlingloff. 2017. Model-based performance prediction by statistical model checking an industrial case study of autonomous transport robots. In Concurrency, Specification and Programming Conference.CEUR."},{"key":"e_1_3_2_14_2","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1145\/3022099.3022101","volume-title":"3rd Workshop on Model-driven Robot Software Engineering","author":"Araiza-Illan Dejanira","year":"2016","unstructured":"Dejanira Araiza-Illan, Anthony G. Pipe, and Kerstin Eder. 2016. Intelligent agent-based stimulation for testing robotic software in human-robot interactions. In 3rd Workshop on Model-driven Robot Software Engineering. ACM, 9\u201316."},{"key":"e_1_3_2_15_2","article-title":"Model-based testing, using belief-desire-intentions agents, of control code for robots in collaborative human-robot interactions","volume":"1","author":"Araiza-Illan Dejanira","year":"2016","unstructured":"Dejanira Araiza-Illan, Tony Pipe, and Kerstin Eder. 2016. Model-based testing, using belief-desire-intentions agents, of control code for robots in collaborative human-robot interactions. arXiv preprint arXiv:1603.00656 1 (2016).","journal-title":"arXiv preprint arXiv:1603.00656"},{"key":"e_1_3_2_16_2","first-page":"23","volume-title":"International Conference on Information Reuse and Integration","author":"Ara\u00fajo Rafael","year":"2017","unstructured":"Rafael Ara\u00fajo, Alexandre Mota, and Sidney Nogueira. 2017. Analyzing cleaning robots using probabilistic model checking. In International Conference on Information Reuse and Integration. Springer, 23\u201351."},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10458-019-09409-x"},{"key":"e_1_3_2_18_2","doi-asserted-by":"crossref","first-page":"6A3\u20131","DOI":"10.1109\/ICNSURV.2017.8011938","volume-title":"Integrated Communications, Navigation and Surveillance Conference (ICNS)","author":"Babiceanu Radu F.","year":"2017","unstructured":"Radu F. Babiceanu and Remzi Seker. 2017. Formal verification of trustworthiness requirements for small unmanned aerial systems. In Integrated Communications, Navigation and Surveillance Conference (ICNS). IEEE, 6A3\u20131."},{"key":"e_1_3_2_19_2","first-page":"57","volume-title":"International Conference on Formal Techniques for Distributed Objects, Components, and Systems","author":"Bao Ran","year":"2019","unstructured":"Ran Bao, Christian Attiogbe, Benoit Delahaye, Paulin Fournier, and Didier Lime. 2019. Parametric statistical model checking of UAV flight plan. In International Conference on Formal Techniques for Distributed Objects, Components, and Systems. Springer, 57\u201374."},{"key":"e_1_3_2_20_2","volume-title":"Conference SIA Simulation Num\u00e9rique","author":"Barbot Beno\u00eet","year":"2017","unstructured":"Beno\u00eet Barbot, B\u00e9atrice B\u00e9rard, Yann Duplouy, and Serge Haddad. 2017. Statistical model-checking for autonomous vehicle safety validation. In Conference SIA Simulation Num\u00e9rique. HAL-Inria."},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-66972-4_15"},{"key":"e_1_3_2_22_2","volume-title":"3rd International Conference on Advances in System Simulation","author":"Behrisch Michael","year":"2011","unstructured":"Michael Behrisch, Laura Bieker, Jakob Erdmann, and Daniel Krajzewicz. 2011. SUMO-Simulation of urban mobility: An overview. In 3rd International Conference on Advances in System Simulation. ThinkMind."},{"key":"e_1_3_2_23_2","first-page":"98","volume-title":"International Joint Conference on Artificial Intelligence","author":"Belardinelli Francesco","year":"2017","unstructured":"Francesco Belardinelli, Panagiotis Kouvaros, and Alessio Lomuscio. 2017. Parameterised verification of data-aware multi-agent systems. In International Joint Conference on Artificial Intelligence. ACM, 98\u2013104."},{"key":"e_1_3_2_24_2","first-page":"91","volume-title":"International Joint Conference on Artificial Intelligence","author":"Belardinelli Francesco","year":"2017","unstructured":"Francesco Belardinelli, Alessio Lomuscio, Aniello Murano, and Sasha Rubin. 2017. Verification of broadcasting multi-agent systems against an epistemic strategy logic. In International Joint Conference on Artificial Intelligence. ACM, Melbourne, Australia, 91\u201397."},{"key":"e_1_3_2_25_2","first-page":"1268","volume-title":"International Conference on Autonomous Agents and Multiagent Systems","author":"Belardinelli Francesco","year":"2017","unstructured":"Francesco Belardinelli, Alessio Lomuscio, Aniello Murano, and Sasha Rubin. 2017. Verification of multi-agent systems with imperfect information and public actions. In International Conference on Autonomous Agents and Multiagent Systems. ACM, 1268\u20131276."},{"key":"e_1_3_2_26_2","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1145\/2970276.2970311","volume-title":"31st IEEE\/ACM International Conference on Automated Software Engineering","author":"Abdessalem Raja Ben","year":"2016","unstructured":"Raja Ben Abdessalem, Shiva Nejati, Lionel C. Briand, and Thomas Stifter. 2016. Testing advanced driver assistance systems using multi-objective search and neural networks. In 31st IEEE\/ACM International Conference on Automated Software Engineering. IEEE\/ACM, New York, NY, 63\u201374."},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1002\/rnc.2914"},{"key":"e_1_3_2_28_2","first-page":"2","volume-title":"IEEE\/ACM 1st International Workshop on Software Engineering for Smart Cyber-physical Systems","author":"Berger Christian","year":"2015","unstructured":"Christian Berger. 2015. Accelerating regression testing for scaled self-driving cars with lightweight virtualization-A case study. In IEEE\/ACM 1st International Workshop on Software Engineering for Smart Cyber-physical Systems. IEEE, 2\u20137."},{"key":"e_1_3_2_29_2","volume-title":"4th European Congress ERTS Embedded Real Time Software (ERTS\u201908)","author":"Berthomieu Bernard","year":"2008","unstructured":"Bernard Berthomieu, Jean-Paul Bodeveix, Patrick Farail, Mamoun Filali, Hubert Garavel, Pierre Gaufillet, Frederic Lang, and Fran\u00e7ois Vernadat. 2008. Fiacre: An intermediate language for model verification in the topcased environment. In 4th European Congress ERTS Embedded Real Time Software (ERTS\u201908). SEE."},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1080\/00207540412331312688"},{"key":"e_1_3_2_31_2","article-title":"Automated search-based robustness testing for autonomous vehicle software","volume":"2016","author":"Betts Kevin M.","year":"2016","unstructured":"Kevin M. Betts and Mikel D. Petty. 2016. Automated search-based robustness testing for autonomous vehicle software. Model. Simul. Eng. 2016 (2016).","journal-title":"Model. Simul. Eng."},{"key":"e_1_3_2_32_2","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1007\/978-3-319-77935-5_2","volume-title":"NASA Formal Methods Symposium","author":"Bhattacharyya Siddhartha","year":"2018","unstructured":"Siddhartha Bhattacharyya, Thomas C. Eskridge, Natasha A. Neogi, Marco Carvalho, and Milton Stafford. 2018. Formal assurance for cooperative intelligent autonomous agents. In NASA Formal Methods Symposium. Springer, 20\u201336."},{"key":"e_1_3_2_33_2","article-title":"A deep learning-based framework for intersectional traffic simulation and editing","volume":"1","author":"Bi Huikun","year":"2019","unstructured":"Huikun Bi, Tianlu Mao, Zhaoqi Wang, and Zhigang Deng. 2019. A deep learning-based framework for intersectional traffic simulation and editing. IEEE Trans. Visualiz. Comput. Graph. 1 (2019).","journal-title":"IEEE Trans. Visualiz. Comput. Graph."},{"key":"e_1_3_2_34_2","article-title":"Testing of RUAV and UGV robots\u2019 collaboration in the Simulink environment","volume":"1","author":"Bicevskis Janis","year":"2013","unstructured":"Janis Bicevskis, Artis Gaujens, and Janis Kalnins. 2013. Testing of RUAV and UGV robots\u2019 collaboration in the Simulink environment. Balt. J. Mod. Comput. 1 (2013).","journal-title":"Balt. J. Mod. Comput."},{"key":"e_1_3_2_35_2","doi-asserted-by":"crossref","first-page":"255","DOI":"10.1007\/978-3-319-11900-7_22","volume-title":"International Conference on Simulation, Modeling, and Programming for Autonomous Robots","author":"Bihlmaier Andreas","year":"2014","unstructured":"Andreas Bihlmaier and Heinz W\u00f6rn. 2014. Robot unit testing. In International Conference on Simulation, Modeling, and Programming for Autonomous Robots. Springer, 255\u2013266."},{"key":"e_1_3_2_36_2","first-page":"139","volume-title":"International Conference on Computer Safety, Reliability, and Security","author":"B\u00f6de Eckard","year":"2018","unstructured":"Eckard B\u00f6de, Matthias B\u00fcker, Ulrich Eberle, Martin Fr\u00e4nzle, Sebastian Gerwinn, and Birte Kramer. 2018. Efficient splitting of test and simulation cases for the verification of highly automated driving functions. In International Conference on Computer Safety, Reliability, and Security. Springer, 139\u2013153."},{"key":"e_1_3_2_37_2","first-page":"1","volume-title":"Symposium on Modeling and Simulation of Complexity in Intelligent, Adaptive and Autonomous Systems","author":"Bohlmann Sebastian","year":"2017","unstructured":"Sebastian Bohlmann, Volkhard Klinger, and Helena Szczerbicka. 2017. Integration of a physical system, machine learning, simulation, validation and control systems towards symbiotic model engineering. In Symposium on Modeling and Simulation of Complexity in Intelligent, Adaptive and Autonomous Systems. ACM, 1\u201312."},{"issue":"4","key":"e_1_3_2_38_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2700318","article-title":"Property-driven design for robot swarms: A design method based on prescriptive modeling and model checking","volume":"9","author":"Brambilla Manuele","year":"2014","unstructured":"Manuele Brambilla, Arne Brutschy, Marco Dorigo, and Mauro Birattari. 2014. Property-driven design for robot swarms: A design method based on prescriptive modeling and model checking. ACM Trans. Auton. Adapt. Syst. 9, 4 (2014), 1\u201328.","journal-title":"ACM Trans. Auton. Adapt. Syst."},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2019.2898267"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10407-2_40"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.micpro.2015.10.006"},{"key":"e_1_3_2_42_2","doi-asserted-by":"crossref","first-page":"307","DOI":"10.1007\/978-3-642-33675-1_27","volume-title":"International Conference on Computer Safety, Reliability, and Security","author":"Brunel Julien","year":"2012","unstructured":"Julien Brunel and Jacques Cazin. 2012. Formal verification of a safety argumentation and application to a complex UAV system. In International Conference on Computer Safety, Reliability, and Security. Springer, 307\u2013318."},{"key":"e_1_3_2_43_2","first-page":"2512","volume-title":"IEEE International Conference on Information and Automation","author":"Bu Qing","year":"2015","unstructured":"Qing Bu, Fuhua Wan, Zhen Xie, Qinhu Ren, Jianhua Zhang, and Sheng Liu. 2015. General simulation platform for vision based UAV testing. In IEEE International Conference on Information and Automation. IEEE, 2512\u20132516."},{"key":"e_1_3_2_44_2","first-page":"265","volume-title":"IEEE Virtual Reality Conference","author":"Burns Don","year":"2004","unstructured":"Don Burns and Robert Osfield. 2004. Tutorial: Open scene graph A: Introduction tutorial: Open scene graph B: Examples and applications. In IEEE Virtual Reality Conference. IEEE, 265\u2013265."},{"key":"e_1_3_2_45_2","article-title":"Users guide version 4.5. 2","volume":"1","author":"CarMaker IPG","year":"2014","unstructured":"IPG CarMaker. 2014. Users guide version 4.5. 2. IPG Automot., Karlsr., Germ. 1 (2014).","journal-title":"IPG Automot., Karlsr., Germ."},{"key":"e_1_3_2_46_2","doi-asserted-by":"crossref","first-page":"1400","DOI":"10.1109\/ROBOT.2007.363180","volume-title":"Proceedings IEEE International Conference on Robotics and Automation","author":"Carpin Stefano","year":"2007","unstructured":"Stefano Carpin, Mike Lewis, Jijun Wang, Stephen Balakirsky, and Chris Scrapper. 2007. USARSim: A robot simulator for research and education. In Proceedings IEEE International Conference on Robotics and Automation. IEEE, 1400\u20131405."},{"key":"e_1_3_2_47_2","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1007\/978-3-030-31157-5_2","volume-title":"International Conference on Tests and Proofs","author":"Cavalcanti Ana","year":"2019","unstructured":"Ana Cavalcanti, James Baxter, Robert M. Hierons, and Raluca Lefticaru. 2019. Testing robots using CSP. In International Conference on Tests and Proofs. Springer, 21\u201338."},{"key":"e_1_3_2_48_2","first-page":"1","volume-title":"International Conference on Integrated Formal Methods","author":"Cavalcanti Ana","year":"2018","unstructured":"Ana Cavalcanti, Alvaro Miyazawa, Augusto Sampaio, Wei Li, Pedro Ribeiro, and Jon Timmis. 2018. Modelling and verification for swarm robotics. In International Conference on Integrated Formal Methods. Springer, 1\u201319."},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.scico.2019.01.004"},{"key":"e_1_3_2_50_2","doi-asserted-by":"crossref","first-page":"8298","DOI":"10.1109\/ICRA.2019.8794430","volume-title":"International Conference on Robotics and Automation (ICRA)","author":"Chao Qianwen","year":"2019","unstructured":"Qianwen Chao, Xiaogang Jin, Hen-Wei Huang, Shaohui Foong, Lap-Fai Yu, and Sai-Kit Yeung. 2019. Force-based heterogeneous traffic simulation for autonomous vehicle testing. In International Conference on Robotics and Automation (ICRA). IEEE, 8298\u20138304."},{"issue":"3","key":"e_1_3_2_51_2","doi-asserted-by":"crossref","first-page":"425","DOI":"10.1109\/TIV.2019.2919470","article-title":"A novel integrated simulation and testing platform for self-driving cars with hardware in the loop","volume":"4","author":"Chen Shitao","year":"2019","unstructured":"Shitao Chen, Yu Chen, Songyi Zhang, and Nanning Zheng. 2019. A novel integrated simulation and testing platform for self-driving cars with hardware in the loop. IEEE Trans. Intell. Vehic. 4, 3 (2019), 425\u2013436.","journal-title":"IEEE Trans. Intell. Vehic."},{"key":"e_1_3_2_52_2","first-page":"949","volume-title":"IEEE Intelligent Vehicles Symposium","author":"Chen Yu","year":"2018","unstructured":"Yu Chen, Shitao Chen, Tangyike Zhang, Songyi Zhang, and Nanning Zheng. 2018. Autonomous vehicle testing and validation platform: Integrated simulation system with hardware in the loop. In IEEE Intelligent Vehicles Symposium. IEEE, 949\u2013956."},{"key":"e_1_3_2_53_2","first-page":"1","volume-title":"OCEANS Conference","author":"Cie\u015blak Patryk","year":"2019","unstructured":"Patryk Cie\u015blak. 2019. Stonefish: An advanced open-source simulation tool designed for marine robotics, with a ROS interface. In OCEANS Conference. IEEE, 1\u20136."},{"key":"e_1_3_2_54_2","first-page":"121","volume-title":"IEEE International Conference on Artificial Intelligence Testing (AITest)","author":"Collet Mathieu","year":"2019","unstructured":"Mathieu Collet, Arnaud Gotlieb, Nadjib Lazaar, and Morten Mossige. 2019. Stress testing of single-arm robots through constraint-based generation of continuous trajectories. In IEEE International Conference on Artificial Intelligence Testing (AITest). IEEE, 121\u2013128."},{"key":"e_1_3_2_55_2","volume-title":"ASME International Mechanical Engineering Congress and Exposition","author":"Compere Marc","year":"2019","unstructured":"Marc Compere, Garrett Holden, Otto Legon, and Roberto Martinez Cruz. 2019. MoVE: A mobility virtual environment for autonomous vehicle testing. In ASME International Mechanical Engineering Congress and Exposition. American Society of Mechanical Engineers."},{"key":"e_1_3_2_56_2","first-page":"1","volume-title":"IEEE 44th International Symposium on Robotics","author":"Cortesi A.","year":"2013","unstructured":"A. Cortesi, P. Ferrara, and N. Chaki. 2013. Static analysis techniques for robotics software verification. In IEEE 44th International Symposium on Robotics. IEEE, 1\u20136."},{"key":"e_1_3_2_57_2","first-page":"471","volume-title":"Federated Conference on Computer Science and Information Systems (FedCSIS)","author":"Cybulski Piotr","year":"2019","unstructured":"Piotr Cybulski. 2019. A framework for autonomous UAV swarm behavior simulation. In Federated Conference on Computer Science and Information Systems (FedCSIS). IEEE, 471\u2013478."},{"key":"e_1_3_2_58_2","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1145\/3194085.3194086","volume-title":"IEEE\/ACM 1st International Workshop on Software Engineering for AI in Autonomous Systems (SEFAIAS)","author":"Damm Werner","year":"2018","unstructured":"Werner Damm and Roland Galbas. 2018. Exploiting learning and scenario-based specification languages for the verification and validation of highly automated driving. In IEEE\/ACM 1st International Workshop on Software Engineering for AI in Autonomous Systems (SEFAIAS). IEEE, 39\u201346."},{"key":"e_1_3_2_59_2","volume-title":"20th Conferencia Iberoamericana en Software Engineering","author":"Neves V\u00e2nia de Oliveira","year":"2019","unstructured":"V\u00e2nia de Oliveira Neves, M\u00e1rcio Eduardo Delamaro, and Paulo Cesar Masiero. 2019. Automated structural software testing of autonomous vehicles. In 20th Conferencia Iberoamericana en Software Engineering. CIbSE."},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/4235.996017"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.robot.2015.11.012"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.21105\/joss.00617"},{"key":"e_1_3_2_63_2","volume-title":"Workshops at the 29th AAAI Conference On Artificial Intelligence (AAAI\u201915)","author":"Dennis Louise A.","year":"2015","unstructured":"Louise A. Dennis, Michael Fisher, and Alan F. T. Winfield. 2015. Towards verifiably ethical robot behaviour. In Workshops at the 29th AAAI Conference On Artificial Intelligence (AAAI\u201915). IEEE."},{"key":"e_1_3_2_64_2","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1007\/978-3-319-67531-2_11","volume-title":"Runtime Verification","author":"Desai Ankush","year":"2017","unstructured":"Ankush Desai, Tommaso Dreossi, and Sanjit A. Seshia. 2017. Combining model checking and runtime verification for safe robotics. In Runtime Verification, Shuvendu Lahiri and Giles Reger (Eds.). Springer International Publishing, Cham, 172\u2013189."},{"key":"e_1_3_2_65_2","volume-title":"21st International Conference on Principles of Distributed Systems (OPODIS\u201917)","author":"Doan Ha Thi Thu","year":"2018","unstructured":"Ha Thi Thu Doan, Fran\u00e7ois Bonnet, and Kazuhiro Ogata. 2018. Model checking of robot gathering. In 21st International Conference on Principles of Distributed Systems (OPODIS\u201917). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik."},{"key":"e_1_3_2_66_2","unstructured":"Simulink Documentation. 2020. Simulation and Model-based Design.Retrieved from: https:\/\/www.mathworks.com\/products\/simulink.html."},{"key":"e_1_3_2_67_2","volume-title":"8th IARP Workshop on Robotics for Risky Environments","author":"Doroftei Daniela","year":"2015","unstructured":"Daniela Doroftei, Anibal Matos, Eduardo Silva, Victor Lobo, Rene Wagemans, and Geert De Cubber. 2015. Operational validation of robots for risky environments. In 8th IARP Workshop on Robotics for Risky Environments. IARP\/EURON."},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.compeleceng.2019.01.016"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11219-016-9322-x"},{"key":"e_1_3_2_70_2","first-page":"1","volume-title":"12th International Conference on Design & Technology of Integrated Systems In Nanoscale Era (DTIS)","author":"Farulla Giuseppe Air\u00f2","year":"2017","unstructured":"Giuseppe Air\u00f2 Farulla and Anna-Lena Lamprecht. 2017. Model checking of security properties: A case study on human-robot interaction processes. In 12th International Conference on Design & Technology of Integrated Systems In Nanoscale Era (DTIS). IEEE, 1\u20136."},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.trc.2019.07.027"},{"key":"e_1_3_2_72_2","article-title":"A rational agent controlling an autonomous vehicle: Implementation and formal verification","volume":"1","author":"Fernandes Lucas E. R.","year":"2017","unstructured":"Lucas E. R. Fernandes, Vinicius Custodio, Gleifer V. Alves, and Michael Fisher. 2017. A rational agent controlling an autonomous vehicle: Implementation and formal verification. arXiv preprint arXiv:1709.02557 1 (2017).","journal-title":"arXiv preprint arXiv:1709.02557"},{"key":"e_1_3_2_73_2","first-page":"263","volume-title":"International Conference on Runtime Verification","author":"Ferrando Angelo","year":"2018","unstructured":"Angelo Ferrando, Louise A. Dennis, Davide Ancona, Michael Fisher, and Viviana Mascardi. 2018. Verifying and validating autonomous systems: Towards an integrated approach. In International Conference on Runtime Verification. Springer, 263\u2013281."},{"key":"e_1_3_2_74_2","first-page":"32","volume-title":"International Conference on Model and Data Engineering","author":"Foughali Mohammed","year":"2019","unstructured":"Mohammed Foughali. 2019. On reconciling schedulability analysis and model checking in robotics. In International Conference on Model and Data Engineering. Springer, 32\u201348."},{"key":"e_1_3_2_75_2","first-page":"2","volume-title":"IEEE\/ACM 6th International FME Workshop on Formal Methods in Software Engineering (FormaliSE)","author":"Foughali Mohammed","year":"2018","unstructured":"Mohammed Foughali, Bernard Berthomieu, Silvano Dal Zilio, Pierre-Emmanuel Hladik, F\u00e9lix Ingrand, and Anthony Mallet. 2018. Formal verification of complex robotic systems on resource-constrained platforms. In IEEE\/ACM 6th International FME Workshop on Formal Methods in Software Engineering (FormaliSE). IEEE, 2\u20139."},{"key":"e_1_3_2_76_2","first-page":"383","volume-title":"International Conference on Formal Engineering Methods","author":"Foughali Mohammed","year":"2016","unstructured":"Mohammed Foughali, Bernard Berthomieu, Silvano Dal Zilio, F\u00e9lix Ingrand, and Anthony Mallet. 2016. Model checking real-time properties on the functional layer of autonomous robots. In International Conference on Formal Engineering Methods. Springer, 383\u2013399."},{"key":"e_1_3_2_77_2","unstructured":"National Science Foundation. 2018. Smart and Autonomous Systems (S&AS) Program Solicitation. Retrieved from https:\/\/www.nsf.gov\/pubs\/2018\/nsf18557\/nsf18557.htm."},{"key":"e_1_3_2_78_2","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1007\/978-3-319-67113-0_8","volume-title":"International Workshop on Formal Methods and Automated Verification of Critical Systems","author":"Gainer Paul","year":"2017","unstructured":"Paul Gainer, Clare Dixon, Kerstin Dautenhahn, Michael Fisher, Ullrich Hustadt, Joe Saunders, and Matt Webster. 2017. CRutoN: Automatic verification of a robotic assistant\u2019s behaviours. In International Workshop on Formal Methods and Automated Verification of Critical Systems. Springer, 119\u2013133."},{"key":"e_1_3_2_79_2","first-page":"318","volume-title":"28th ACM SIGSOFT International Symposium on Software Testing and Analysis","author":"Gambi Alessio","year":"2019","unstructured":"Alessio Gambi, Marc Mueller, and Gordon Fraser. 2019. Automatically testing self-driving cars with search-based procedural content generation. In 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, 318\u2013328."},{"key":"e_1_3_2_80_2","unstructured":"Shenjian Gao and Yanwen Tan. 2017. Paving the Way for Self-driving Cars - Software Testing for Safety-critical Systems Based on Machine Learning: A Systematic Mapping Study and a Survey. Blekinge Tekniska H\u00f6gsola."},{"key":"e_1_3_2_81_2","doi-asserted-by":"crossref","first-page":"803","DOI":"10.1109\/ICARCV.2018.8581290","volume-title":"15th International Conference on Control, Automation, Robotics and Vision (ICARCV)","author":"Garz\u00f3n Mario","year":"2018","unstructured":"Mario Garz\u00f3n and Anne Spalanzani. 2018. An hybrid simulation tool for autonomous cars in very high traffic scenarios. In 15th International Conference on Control, Automation, Robotics and Vision (ICARCV). IEEE, 803\u2013808."},{"key":"e_1_3_2_82_2","first-page":"45","volume-title":"International Conference on Computer Safety, Reliability, and Security","author":"Gauerhof Lydia","year":"2018","unstructured":"Lydia Gauerhof, Peter Munk, and Simon Burton. 2018. Structuring validation targets of a machine learning function applied to automated driving. In International Conference on Computer Safety, Reliability, and Security. Springer, 45\u201358."},{"issue":"1","key":"e_1_3_2_83_2","doi-asserted-by":"crossref","first-page":"9248","DOI":"10.1016\/j.ifacol.2017.08.1287","article-title":"Parametric formal verification: The robotic paint spraying case study","volume":"50","author":"Geretti Luca","year":"2017","unstructured":"Luca Geretti, Riccardo Muradore, Davide Bresolin, Paolo Fiorini, and Tiziano Villa. 2017. Parametric formal verification: The robotic paint spraying case study. IFAC-PapersOnLine 50, 1 (2017), 9248\u20139253.","journal-title":"IFAC-PapersOnLine"},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.procir.2019.04.269"},{"key":"e_1_3_2_85_2","doi-asserted-by":"publisher","DOI":"10.1016\/0164-1212(90)90074-V"},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-54862-8_13"},{"key":"e_1_3_2_87_2","first-page":"54","volume-title":"International Conference on Formal Engineering Methods","author":"Gjondrekaj Edmond","year":"2012","unstructured":"Edmond Gjondrekaj, Michele Loreti, Rosario Pugliese, Francesco Tiezzi, Carlo Pinciroli, Manuele Brambilla, Mauro Birattari, and Marco Dorigo. 2012. Towards a formal verification methodology for collective robotic systems. In International Conference on Formal Engineering Methods. Springer, 54\u201370."},{"key":"e_1_3_2_88_2","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1109\/ASE.2019.00013","volume-title":"34th IEEE\/ACM International Conference on Automated Software Engineering (ASE)","author":"Gladisch Christoph","year":"2019","unstructured":"Christoph Gladisch, Thomas Heinz, Christian Heinzemann, Jens Oehlerking, Anne von Vietinghoff, and Tim Pfitzer. 2019. Experience paper: Search-based testing in automated driving control applications. In 34th IEEE\/ACM International Conference on Automated Software Engineering (ASE). IEEE, 26\u201337."},{"key":"e_1_3_2_89_2","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1007\/978-3-030-30446-1_5","volume-title":"International Conference on Software Engineering and Formal Methods","author":"Gleirscher Mario","year":"2019","unstructured":"Mario Gleirscher, Simon Foster, and Yakoub Nemouchi. 2019. Evolution of formal model-based assurance cases for autonomous robots. In International Conference on Software Engineering and Formal Methods. Springer, 87\u2013104."},{"key":"e_1_3_2_90_2","doi-asserted-by":"publisher","DOI":"10.1145\/3357231"},{"key":"e_1_3_2_91_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-15024-6_10"},{"key":"e_1_3_2_92_2","first-page":"1708","volume-title":"21st International Conference on Intelligent Transportation Systems (ITSC)","author":"Gruber Felix","year":"2018","unstructured":"Felix Gruber and Matthias Althoff. 2018. Anytime safety verification of autonomous vehicles. In 21st International Conference on Intelligent Transportation Systems (ITSC). IEEE, 1708\u20131714."},{"key":"e_1_3_2_93_2","article-title":"eBotworks: A software platform for developing and evaluating communicative autonomous systems","volume":"1","author":"Gupta K. M.","year":"2015","unstructured":"K. M. Gupta and K. Gillespie. 2015. eBotworks: A software platform for developing and evaluating communicative autonomous systems. AUVSI Unman. Syst., Atlanta, GA 1 (2015).","journal-title":"AUVSI Unman. Syst., Atlanta, GA"},{"key":"e_1_3_2_94_2","first-page":"26","volume-title":"Cybersecurity Symposium (CYBERSEC)","author":"Hagerman Seana","year":"2016","unstructured":"Seana Hagerman, Anneliese Andrews, and Stephen Oakes. 2016. Security testing of an unmanned aerial vehicle (UAV). In Cybersecurity Symposium (CYBERSEC). IEEE, 26\u201331."},{"key":"e_1_3_2_95_2","first-page":"44","volume-title":"IEEE\/ACM 5th International FME Workshop on Formal Methods in Software Engineering (FormaliSE)","author":"Halder Raju","year":"2017","unstructured":"Raju Halder, Jos\u00e9 Proen\u00e7a, Nuno Macedo, and Andr\u00e9 Santos. 2017. Formal verification of ROS-based robotic applications using timed-automata. In IEEE\/ACM 5th International FME Workshop on Formal Methods in Software Engineering (FormaliSE). IEEE, 44\u201350."},{"key":"e_1_3_2_96_2","first-page":"906","volume-title":"12th International Conference on Developments in eSystems Engineering (DeSE)","author":"Heikkinen Jani Erik","year":"2019","unstructured":"Jani Erik Heikkinen, Salimzhan Gafurov, Sergey Kopylov, Tatiana Minav, Sergey Grebennikov, and Artur Kurbanov. 2019. Hardware-in-the-loop platform for testing autonomous vehicle control algorithms. In 12th International Conference on Developments in eSystems Engineering (DeSE). IEEE, 906\u2013911."},{"key":"e_1_3_2_97_2","volume-title":"Tools for Constructing Requirements Specification: The SCR Toolset at the Age of Ten","author":"Heitmeyer Constance","year":"2005","unstructured":"Constance Heitmeyer, Myla Archer, Ramesh Bharadwaj, and Ralph Jeffords. 2005. Tools for Constructing Requirements Specification: The SCR Toolset at the Age of Ten. Technical Report. Naval Research Lab Washington DC Center for High Assurance Computing Systems."},{"key":"e_1_3_2_98_2","article-title":"Software cost reduction","volume":"1","author":"Heitmeyer Constance L.","year":"2002","unstructured":"Constance L. Heitmeyer. 2002. Software cost reduction. Encyc. Softw. Eng. 1 (2002).","journal-title":"Encyc. Softw. Eng."},{"key":"e_1_3_2_99_2","first-page":"54","volume-title":"IEEE\/ACM 3rd FME Workshop on Formal Methods in Software Engineering","author":"Heitmeyer Constance L.","year":"2015","unstructured":"Constance L. Heitmeyer and Elizabeth I. Leonard. 2015. Obtaining trust in autonomous systems: Tools for formal model synthesis and validation. In IEEE\/ACM 3rd FME Workshop on Formal Methods in Software Engineering. IEEE, 54\u201360."},{"key":"e_1_3_2_100_2","doi-asserted-by":"publisher","DOI":"10.1002\/j.2334-5837.2016.00179.x"},{"key":"e_1_3_2_101_2","first-page":"719","volume-title":"IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)","author":"Hereid Ayonga","year":"2017","unstructured":"Ayonga Hereid and Aaron D. Ames. 2017. FROST*: Fast robot optimization and simulation toolkit. In IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 719\u2013726."},{"key":"e_1_3_2_102_2","doi-asserted-by":"publisher","DOI":"10.1145\/359576.359585"},{"key":"e_1_3_2_103_2","doi-asserted-by":"crossref","first-page":"247","DOI":"10.1007\/978-3-319-11164-3_20","volume-title":"International Conference on Runtime Verification","author":"Huang Jeff","year":"2014","unstructured":"Jeff Huang, Cansu Erdogan, Yi Zhang, Brandon Moore, Qingzhou Luo, Aravind Sundaresan, and Grigore Rosu. 2014. ROSRV: Runtime verification for robots. In International Conference on Runtime Verification. Springer, 247\u2013254."},{"key":"e_1_3_2_104_2","first-page":"69","article-title":"Model checking for verification in UAV cooperative control applications","volume":"444","author":"Humphrey Laura R.","year":"2013","unstructured":"Laura R. Humphrey. 2013. Model checking for verification in UAV cooperative control applications. Rec. Adv. Res. Unman. Aer. Vehic. 444 (2013), 69\u2013117.","journal-title":"Rec. Adv. Res. Unman. Aer. Vehic."},{"key":"e_1_3_2_105_2","article-title":"Trafficware SYNCHRO 6 user guide","volume":"11","author":"Husch David","year":"2004","unstructured":"David Husch and John Albeck. 2004. Trafficware SYNCHRO 6 user guide. TrafficWare, Alb., Calif. 11 (2004).","journal-title":"TrafficWare, Alb., Calif."},{"key":"e_1_3_2_106_2","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1145\/2377576.2377606","volume-title":"10th Performance Metrics for Intelligent Systems Workshop","author":"Jacoff Adam","year":"2010","unstructured":"Adam Jacoff, Hui-Min Huang, Elena Messina, Ann Virts, and Anthony Downs. 2010. Comprehensive standard test suites for the performance evaluation of mobile robots. In 10th Performance Metrics for Intelligent Systems Workshop. ACM, 161\u2013168."},{"key":"e_1_3_2_107_2","volume-title":"6th International Modelica Conference","author":"Junghanns Andreas","year":"2008","unstructured":"Andreas Junghanns, Jakob Mauss, Mugur Tatar, et\u00a0al. 2008. TestWeaver\u2014A tool for simulation-based test of mechatronic designs. In 6th International Modelica Conference. Citeseer."},{"key":"e_1_3_2_108_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.scico.2017.05.006"},{"key":"e_1_3_2_109_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIV.2018.2886678"},{"key":"e_1_3_2_110_2","first-page":"287","volume-title":"ACM\/IEEE 9th International Conference on Cyber-Physical Systems (ICCPS)","author":"Kato Shinpei","year":"2018","unstructured":"Shinpei Kato, Shota Tokunaga, Yuya Maruyama, Seiya Maeda, Manato Hirabayashi, Yuki Kitsukawa, Abraham Monrroy, Tomohito Ando, Yusuke Fujii, and Takuya Azumi. 2018. Autoware on board: Enabling autonomous vehicles with embedded systems. In ACM\/IEEE 9th International Conference on Cyber-Physical Systems (ICCPS). IEEE, 287\u2013296."},{"key":"e_1_3_2_111_2","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1145\/3243127.3243129","volume-title":"1st International Workshop on Machine Learning and Software Engineering in Symbiosis","author":"Khosrowjerdi Hojat","year":"2018","unstructured":"Hojat Khosrowjerdi and Karl Meinke. 2018. Learning-based testing for autonomous systems using spatial and temporal requirements. In 1st International Workshop on Machine Learning and Software Engineering in Symbiosis. ACM, 6\u201315."},{"issue":"1","key":"e_1_3_2_112_2","first-page":"5","article-title":"Testing autonomous vehicle software in the virtual prototyping environment","volume":"9","author":"Kim Baekgyu","year":"2016","unstructured":"Baekgyu Kim, Yusuke Kashiba, Siyuan Dai, and Shinichi Shiraishi. 2016. Testing autonomous vehicle software in the virtual prototyping environment. IEEE Embed. Syst. Lett. 9, 1 (2016), 5\u20138.","journal-title":"IEEE Embed. Syst. Lett."},{"key":"e_1_3_2_113_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00108"},{"key":"e_1_3_2_114_2","doi-asserted-by":"crossref","first-page":"418","DOI":"10.1109\/QRS.2019.00058","volume-title":"IEEE 19th International Conference on Software Quality, Reliability and Security (QRS)","author":"Kl\u00fcck Florian","year":"2019","unstructured":"Florian Kl\u00fcck, Martin Zimmermann, Franz Wotawa, and Mihai Nica. 2019. Genetic algorithm-based test parameter optimization for ADAS system testing. In IEEE 19th International Conference on Software Quality, Reliability and Security (QRS). IEEE, 418\u2013425."},{"key":"e_1_3_2_115_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-C.2017.67"},{"key":"e_1_3_2_116_2","first-page":"2149","volume-title":"IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)","author":"Koenig Nathan","year":"2004","unstructured":"Nathan Koenig and Andrew Howard. 2004. Design and use paradigms for Gazebo, an open-source multi-robot simulator. In IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2149\u20132154."},{"key":"e_1_3_2_117_2","first-page":"200","volume-title":"International Conference on TOOLS and Algorithms for the Construction and Analysis of Systems","author":"Kong Soonho","year":"2015","unstructured":"Soonho Kong, Sicun Gao, Wei Chen, and Edmund Clarke. 2015. dReach: \\(\\delta\\) -reachability analysis for hybrid systems. In International Conference on TOOLS and Algorithms for the Construction and Analysis of Systems. Springer, 200\u2013205."},{"key":"e_1_3_2_118_2","doi-asserted-by":"crossref","first-page":"604","DOI":"10.1109\/ICRA.2019.8793875","volume-title":"International Conference on Robotics and Automation (ICRA)","author":"Koolen Twan","year":"2019","unstructured":"Twan Koolen and Robin Deits. 2019. Julia for robotics: Simulation and real-time control in a high-level programming language. In International Conference on Robotics and Automation (ICRA). IEEE, 604\u2013611."},{"key":"e_1_3_2_119_2","doi-asserted-by":"publisher","DOI":"10.4271\/2016-01-0128"},{"key":"e_1_3_2_120_2","volume-title":"AAAI Conference on Artificial Intelligence","author":"Kouvaros Panagiotis","year":"2015","unstructured":"Panagiotis Kouvaros and Alessio Lomuscio. 2015. A counter abstraction technique for the verification of robot swarms. In AAAI Conference on Artificial Intelligence. AAAI PRESS."},{"key":"e_1_3_2_121_2","doi-asserted-by":"publisher","DOI":"10.5555\/2832249.2832399"},{"key":"e_1_3_2_122_2","first-page":"1200","volume-title":"International Conference on Autonomous Agents & Multiagent Systems","author":"Kouvaros Panagiotis","year":"2016","unstructured":"Panagiotis Kouvaros and Alessio Lomuscio. 2016. Formal verification of opinion formation in swarms. In International Conference on Autonomous Agents & Multiagent Systems. ACM, 1200\u20131208."},{"key":"e_1_3_2_123_2","doi-asserted-by":"publisher","DOI":"10.5555\/3306127.3331691"},{"key":"e_1_3_2_124_2","doi-asserted-by":"crossref","first-page":"585","DOI":"10.1007\/978-3-642-22110-1_47","volume-title":"International Conference on Computer-aided Verification","author":"Kwiatkowska Marta","year":"2011","unstructured":"Marta Kwiatkowska, Gethin Norman, and David Parker. 2011. PRISM 4.0: Verification of probabilistic real-time systems. In International Conference on Computer-aided Verification. Springer, 585\u2013591."},{"key":"e_1_3_2_125_2","unstructured":"John E. Laird. 2019. The Soar cognitive architecture. MIT Press."},{"key":"e_1_3_2_126_2","doi-asserted-by":"publisher","DOI":"10.1007\/s100090050010"},{"key":"e_1_3_2_127_2","first-page":"1083","volume-title":"IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)","author":"Lasbouygues Adrien","year":"2015","unstructured":"Adrien Lasbouygues, Benoit Ropars, Robin Passama, David Andreu, and Lionel Lapierre. 2015. Atoms based control of mobile robots with hardware-in-the-loop validation. In IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 1083\u20131090."},{"key":"e_1_3_2_128_2","first-page":"1842","volume-title":"IEEE\/RSJ International Conference on Intelligent Robots and Systems","author":"Laval Jannik","year":"2013","unstructured":"Jannik Laval, Luc Fabresse, and Noury Bouraqadi. 2013. A methodology for testing mobile autonomous robots. In IEEE\/RSJ International Conference on Intelligent Robots and Systems. IEEE, 1842\u20131847."},{"key":"e_1_3_2_129_2","first-page":"433","volume-title":"IEEE International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE Conference on Robotics, Automation and Mechatronics (RAM)","author":"Ledent Philippe","year":"2019","unstructured":"Philippe Ledent, Anshul Paigwar, Alessandro Renzaglia, Radu Mateescu, and Christian Laugier. 2019. Formal validation of probabilistic collision risk estimation for autonomous driving. In IEEE International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE Conference on Robotics, Automation and Mechatronics (RAM). IEEE, 433\u2013438."},{"key":"e_1_3_2_130_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIV.2016.2608003"},{"key":"e_1_3_2_131_2","first-page":"727","volume-title":"IEEE 55th Conference on Decision and Control (CDC)","author":"Li Nan","year":"2016","unstructured":"Nan Li, Dave Oyler, Mengxuan Zhang, Yildiray Yildiz, Anouck Girard, and Ilya Kolmanovsky. 2016. Hierarchical reasoning game theory based approach for evaluation and testing of autonomous vehicle control systems. In IEEE 55th Conference on Decision and Control (CDC). IEEE, 727\u2013733."},{"issue":"5","key":"e_1_3_2_132_2","first-page":"1782","article-title":"Game theoretic modeling of driver and vehicle interactions for verification and validation of autonomous vehicle control systems","volume":"26","author":"Li Nan","year":"2017","unstructured":"Nan Li, Dave W. Oyler, Mengxuan Zhang, Yildiray Yildiz, Ilya Kolmanovsky, and Anouck R. Girard. 2017. Game theoretic modeling of driver and vehicle interactions for verification and validation of autonomous vehicle control systems. IEEE Trans. Contr. Syst. Technol. 26, 5 (2017), 1782\u20131797.","journal-title":"IEEE Trans. Contr. Syst. Technol."},{"key":"e_1_3_2_133_2","first-page":"287","volume-title":"9th International Conference on Software Engineering and Applications (ICSOFT-EA)","author":"Lill Raimar","year":"2014","unstructured":"Raimar Lill and Francesca Saglietti. 2014. Testing the cooperation of autonomous robotic agents. In 9th International Conference on Software Engineering and Applications (ICSOFT-EA). IEEE, 287\u2013296."},{"key":"e_1_3_2_134_2","first-page":"35","volume-title":"IEEE\/ACM 2nd International Workshop on Metamorphic Testing (MET)","author":"Lindvall Mikael","year":"2017","unstructured":"Mikael Lindvall, Adam Porter, Gudjon Magnusson, and Christoph Schulze. 2017. Metamorphic model-based testing of autonomous systems. In IEEE\/ACM 2nd International Workshop on Metamorphic Testing (MET). IEEE, 35\u201341."},{"key":"e_1_3_2_135_2","first-page":"189","volume-title":"International Conference on Autonomous Agents and Multiagent Systems","author":"Lomuscio Alessio","year":"2015","unstructured":"Alessio Lomuscio and Jakub Michaliszyn. 2015. Verifying multi-agent systems by model checking three-valued abstractions. In International Conference on Autonomous Agents and Multiagent Systems. ACM, 189\u2013198."},{"key":"e_1_3_2_136_2","first-page":"161","volume-title":"International Conference on Autonomous Agents and Multiagent Systems","author":"Lomuscio Alessio","year":"2019","unstructured":"Alessio Lomuscio and Edoardo Pirovano. 2019. A counter abstraction technique for the verification of probabilistic swarm systems. In International Conference on Autonomous Agents and Multiagent Systems. ACM, 161\u2013169."},{"key":"e_1_3_2_137_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10009-015-0378-x"},{"key":"e_1_3_2_138_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ifacol.2016.10.332"},{"key":"e_1_3_2_139_2","doi-asserted-by":"publisher","DOI":"10.1145\/3342355"},{"issue":"3","key":"e_1_3_2_140_2","doi-asserted-by":"crossref","first-page":"713","DOI":"10.3182\/20140824-6-ZA-1003.02809","article-title":"The MAV3DSim: A simulation platform for research, education and validation of UAV controllers","volume":"47","author":"Lugo-C\u00e1rdenas Israel","year":"2014","unstructured":"Israel Lugo-C\u00e1rdenas, Gerardo Flores, and Rogelio Lozano. 2014. The MAV3DSim: A simulation platform for research, education and validation of UAV controllers. IFAC Proc. 47, 3 (2014), 713\u2013717.","journal-title":"IFAC Proc."},{"key":"e_1_3_2_141_2","first-page":"204","volume-title":"IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC)","author":"Luo Chenxia","year":"2018","unstructured":"Chenxia Luo, Rui Wang, Yu Jiang, Kang Yang, Yong Guan, Xiaojuan Li, and Zhiping Shi. 2018. Runtime verification of robots collision avoidance case study. In IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC). IEEE, 204\u2013212."},{"key":"e_1_3_2_142_2","first-page":"56","volume-title":"IEEE 27th International Conference on Tools with Artificial Intelligence (ICTAI)","author":"Lyons Damian M.","year":"2015","unstructured":"Damian M. Lyons, Ronald C. Arkin, Shu Jiang, Dagan Harrington, Feng Tang, and Peng Tang. 2015. Probabilistic verification of multi-robot missions in uncertain environments. In IEEE 27th International Conference on Tools with Artificial Intelligence (ICTAI). IEEE, 56\u201363."},{"key":"e_1_3_2_143_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.robot.2017.07.001"},{"key":"e_1_3_2_144_2","first-page":"89","volume-title":"ACM\/IEEE 22nd International Conference on Model Driven Engineering Languages and Systems (MODELS)","author":"Majzik Istv\u00e1n","year":"2019","unstructured":"Istv\u00e1n Majzik, Oszk\u00e1r Semer\u00e1th, Csaba Hajdu, Krist\u00f3f Marussy, Zolt\u00e1n Szatm\u00e1ri, Zolt\u00e1n Micskei, Andr\u00e1s V\u00f6r\u00f6s, Aren A. Babikian, and D\u00e1niel Varr\u00f3. 2019. Towards system-level testing with coverage guarantees for autonomous vehicles. In ACM\/IEEE 22nd International Conference on Model Driven Engineering Languages and Systems (MODELS). IEEE, 89\u201394."},{"key":"e_1_3_2_145_2","doi-asserted-by":"crossref","first-page":"152","DOI":"10.1007\/978-3-540-30206-3_12","volume-title":"Formal Techniques, Modelling and Analysis of Timed and Fault-tolerant Systems","author":"Maler Oded","year":"2004","unstructured":"Oded Maler and Dejan Nickovic. 2004. Monitoring temporal properties of continuous signals. In Formal Techniques, Modelling and Analysis of Timed and Fault-tolerant Systems. Springer, 152\u2013166."},{"key":"e_1_3_2_146_2","first-page":"4627","volume-title":"IEEE International Conference on Robotics and Automation","author":"Mallet Anthony","year":"2010","unstructured":"Anthony Mallet, C\u00e9dric Pasteur, Matthieu Herrb, S\u00e9verin Lemaignan, and F\u00e9lix Ingrand. 2010. GenoM3: Building middleware-independent robotic components. In IEEE International Conference on Robotics and Automation. IEEE, 4627\u20134632."},{"key":"e_1_3_2_147_2","first-page":"701","volume-title":"IEEE International Conference on Systems, Man, and Cybernetics","author":"Mamrot Michel","year":"2015","unstructured":"Michel Mamrot, Stefan Marchlewitz, Jan-Peter Nicklas, Petra Winzer, Thomas Tetzlaff, Philipp Kemper, and Ulf Witkowski. 2015. Model-based test and validation support for autonomous mechatronic systems. In IEEE International Conference on Systems, Man, and Cybernetics. IEEE, Hong Kong, 701\u2013706."},{"key":"e_1_3_2_148_2","first-page":"1","volume-title":"ACM Workshop on Domain-specific Modeling","author":"Mamun Md Abdullah Al","year":"2013","unstructured":"Md Abdullah Al Mamun, Christian Berger, and Jorgen Hansson. 2013. MDE-based sensor management and verification for a self-driving miniature vehicle. In ACM Workshop on Domain-specific Modeling. ACM, 1\u20136."},{"key":"e_1_3_2_149_2","first-page":"1","volume-title":"MTS\/IEEE OCEANS Conference","author":"Manh\u00e3es Musa Morena Marcusso","year":"2016","unstructured":"Musa Morena Marcusso Manh\u00e3es, Sebastian A. Scherer, Martin Voss, Luiz Ricardo Douat, and Thomas Rauschenbach. 2016. UUV simulator: A Gazebo-based package for underwater intervention and multi-robot simulation. In MTS\/IEEE OCEANS Conference. IEEE, 1\u20138."},{"key":"e_1_3_2_150_2","first-page":"785","volume-title":"26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering","author":"Mansoor Niloofar","year":"2018","unstructured":"Niloofar Mansoor, Jonathan A. Saddler, Bruno Silva, Hamid Bagheri, Myra B. Cohen, and Shane Farritor. 2018. Modeling and testing a family of surgical robots: An experience report. In 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 785\u2013790."},{"key":"e_1_3_2_151_2","first-page":"1","volume-title":"28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)","author":"Mariager Casper Sloth","year":"2019","unstructured":"Casper Sloth Mariager, Daniel Kjaer Bonde Fischer, Jakob Kristiansen, and Matthias Rehm. 2019. Co-designing and field-testing adaptable robots for triggering positive social interactions for adolescents with cerebral palsy. In 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN). IEEE, 1\u20136."},{"key":"e_1_3_2_152_2","volume-title":"version 7.10.0 (R2010a)","year":"2010","unstructured":"MATLAB. 2010. version 7.10.0 (R2010a). The MathWorks Inc., Natick, Massachusetts."},{"key":"e_1_3_2_153_2","first-page":"1","volume-title":"Artificial Intelligence Safety Conference","author":"McDermid John Alexander","year":"2019","unstructured":"John Alexander McDermid, Yan Jia, and Ibrahim Habli. 2019. Towards a framework for safety assurance of autonomous systems. In Artificial Intelligence Safety Conference. CEUR Workshop Proceedings, 1\u20137."},{"key":"e_1_3_2_154_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2019.2894381"},{"key":"e_1_3_2_155_2","article-title":"Hypothesis","author":"McLver David R.","year":"2021","unstructured":"David R. McLver. 2021. Hypothesis. Retrieved from https:\/\/github.com\/HypothesisWorks\/hypothesis.","journal-title":"Retrieved from https:\/\/github.com\/HypothesisWorks\/hypothesis"},{"key":"e_1_3_2_156_2","first-page":"1","volume-title":"SoutheastCon","author":"Medrano-Berumen Christopher","year":"2019","unstructured":"Christopher Medrano-Berumen and Mustafaff Ilhan Akba\u015f. 2019. Abstract simulation scenario generation for autonomous vehicle verification. In SoutheastCon. IEEE, 1\u20136."},{"key":"e_1_3_2_157_2","unstructured":"Alvaro Miyazawa Pedro Ribeiro Wei Li A. L. C. Cavalcanti Jon Timmis and J. C. P. Woodcock. 2016. RoboChart: A state-machine notation for modelling and verification of mobile and autonomous robots. (2016)."},{"key":"e_1_3_2_158_2","first-page":"452","volume-title":"4th International Conference on System Reliability and Safety (ICSRS)","author":"Mongelli Maurizio","year":"2019","unstructured":"Maurizio Mongelli, Marco Muselli, Andrea Scorzoni, and Enrico Ferrari. 2019. Accelerating prism validation of vehicle platooning through machine learning. In 4th International Conference on System Reliability and Safety (ICSRS). IEEE, 452\u2013456."},{"key":"e_1_3_2_159_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10846-018-0901-x"},{"key":"e_1_3_2_160_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2017.10.031"},{"key":"e_1_3_2_161_2","volume-title":"IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)","author":"Munawar Adnan","year":"2019","unstructured":"Adnan Munawar and Gregory S. Fischer. 2019. An asynchronous multi-body simulation framework for real-time dynamics, haptics and learning with application to surgical robots. In IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE."},{"key":"e_1_3_2_162_2","first-page":"269","volume-title":"18th IEEE International Conference and Workshops on Engineering of Computer-based Systems","author":"Mutter Florian","year":"2011","unstructured":"Florian Mutter, Stefanie Gareis, Bernhard Sch\u00e4tz, Andreas Bayha, Franziska Gr\u00fcneis, Michael Kanis, and Dagmar Koss. 2011. Model-driven in-the-loop validation: Simulation-based testing of UAV software using virtual environments. In 18th IEEE International Conference and Workshops on Engineering of Computer-based Systems. IEEE, 269\u2013275."},{"key":"e_1_3_2_163_2","first-page":"1873","volume-title":"Human Factors and Ergonomics Society Annual Meeting","author":"Naujoks Frederik","year":"2018","unstructured":"Frederik Naujoks, Sebastian Hergeth, Katharina Wiedemann, Nadja Sch\u00f6mig, and Andreas Keinath. 2018. Use cases for assessing, testing, and validating the human machine interface of automated driving systems. In Human Factors and Ergonomics Society Annual Meeting. SAGE Publications, 1873\u20131877."},{"key":"e_1_3_2_164_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10458-011-9175-4"},{"key":"e_1_3_2_165_2","unstructured":"Royal Academy of Engineering. 2015. Innovation in autonomous systems: Summary of an event held on Monday 22 June 2015 at the Royal Academy of Engineering."},{"key":"e_1_3_2_166_2","volume-title":"SAE World Congress and Exhibition","author":"O\u2019Kelly Matthew","year":"2016","unstructured":"Matthew O\u2019Kelly, Houssam Abbas, Sicun Gao, Shin\u2019ichi Shiraishi, Shinpei Kato, and Rahul Mangharam. 2016. APEX: Autonomous vehicle plan verification and execution. In SAE World Congress and Exhibition. SAE International."},{"key":"e_1_3_2_167_2","first-page":"9827","volume-title":"Conference on Advances in Neural Information Processing Systems","author":"O\u2019Kelly Matthew","year":"2018","unstructured":"Matthew O\u2019Kelly, Aman Sinha, Hongseok Namkoong, Russ Tedrake, and John C. Duchi. 2018. Scalable end-to-end autonomous vehicle testing via rare-event simulation. In Conference on Advances in Neural Information Processing Systems. NeurIPS, 9827\u20139838."},{"key":"e_1_3_2_168_2","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1145\/3022099.3022104","volume-title":"3rd Workshop on Model-driven Robot Software Engineering","author":"Opfer Stephan","year":"2016","unstructured":"Stephan Opfer, Stefan Niemczyk, and Kurt Geihs. 2016. Multi-agent plan verification with answer set programming. In 3rd Workshop on Model-driven Robot Software Engineering. ACM, 32\u201339."},{"key":"e_1_3_2_169_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-11900-7_39"},{"issue":"20","key":"e_1_3_2_170_2","doi-asserted-by":"crossref","first-page":"4456","DOI":"10.3390\/s19204456","article-title":"Multiple event-based simulation scenario generation approach for autonomous vehicle smart sensors and devices","volume":"19","author":"Park Jisun","year":"2019","unstructured":"Jisun Park, Mingyun Wen, Yunsick Sung, and Kyungeun Cho. 2019. Multiple event-based simulation scenario generation approach for autonomous vehicle smart sensors and devices. Sensors 19, 20 (2019), 4456.","journal-title":"Sensors"},{"key":"e_1_3_2_171_2","article-title":"Compositional verification for autonomous systems with deep learning components","volume":"1","author":"Pasareanu Corina S.","year":"2018","unstructured":"Corina S. Pasareanu, Divya Gopinath, and Huafeng Yu. 2018. Compositional verification for autonomous systems with deep learning components. 1 (2018). arXiv preprint arXiv:1810.08303","journal-title":"arXiv preprint arXiv:1810.08303"},{"key":"e_1_3_2_172_2","doi-asserted-by":"crossref","first-page":"3399","DOI":"10.1109\/SMC.2014.6974453","volume-title":"IEEE International Conference on Systems, Man, and Cybernetics (SMC)","author":"Pathak Shashank","year":"2014","unstructured":"Shashank Pathak, Giorgio Metta, and Armando Tacchella. 2014. Is verification a requisite for safe adaptive robots? In IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 3399\u20133402."},{"key":"e_1_3_2_173_2","doi-asserted-by":"crossref","first-page":"286","DOI":"10.1145\/2245276.2245333","volume-title":"27th Annual ACM Symposium on Applied Computing","author":"Pereira Jos\u00e9 L. F.","year":"2012","unstructured":"Jos\u00e9 L. F. Pereira and Rosaldo J. F. Rossetti. 2012. An integrated architecture for autonomous vehicles simulation. In 27th Annual ACM Symposium on Applied Computing. ACM, 286\u2013292."},{"key":"e_1_3_2_174_2","unstructured":"Mauro Pezze and Michal Young. 2007. In Software Testing and Analysis: Process Principles and Techniques . Wiley."},{"key":"e_1_3_2_175_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11277-013-1107-y"},{"key":"e_1_3_2_176_2","first-page":"75","volume-title":"31st Annual ACM Symposium on User Interface Software and Technology","author":"Porfirio David","year":"2018","unstructured":"David Porfirio, Allison Saupp\u00e9, Aws Albarghouthi, and Bilge Mutlu. 2018. Authoring and verifying human-robot interactions. In 31st Annual ACM Symposium on User Interface Software and Technology. ACM, 75\u201386."},{"key":"e_1_3_2_177_2","first-page":"317","volume-title":"Annual Conference on Artificial Intelligence","author":"Proetzsch Martin","year":"2010","unstructured":"Martin Proetzsch, Fabian Zimmermann, Robert Eschbach, Johannes Kloos, and Karsten Berns. 2010. A systematic testing approach for autonomous mobile robots using domain-specific languages. In Annual Conference on Artificial Intelligence. Springer, 317\u2013324."},{"key":"e_1_3_2_178_2","volume-title":"ICRA Workshop on Open Source Software","author":"Quigley Morgan","year":"2009","unstructured":"Morgan Quigley, Ken Conley, Brian Gerkey, Josh Faust, Tully Foote, Jeremy Leibs, Rob Wheeler, and Andrew Y. Ng. 2009. ROS: An open-source Robot Operating System. In ICRA Workshop on Open Source Software."},{"key":"e_1_3_2_179_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.3048047"},{"key":"e_1_3_2_180_2","first-page":"786","volume-title":"Design, Automation & Test in Europe Conference & Exhibition (DATE)","author":"Ramanathan Arvind","year":"2016","unstructured":"Arvind Ramanathan, Laura L. Pullum, Faraz Hussain, Dwaipayan Chakrabarty, and Sumit Kumar Jha. 2016. Integrating symbolic and statistical methods for testing intelligent systems: Applications to machine learning and computer vision. In Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 786\u2013791."},{"key":"e_1_3_2_181_2","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1145\/3194085.3194087","volume-title":"IEEE\/ACM 1st International Workshop on Software Engineering for AI in Autonomous Systems (SEFAIAS)","author":"Rao Q.","year":"2018","unstructured":"Q. Rao and J. Frtunikj. 2018. Deep learning for self-driving cars: Chances and challenges. In IEEE\/ACM 1st International Workshop on Software Engineering for AI in Autonomous Systems (SEFAIAS). IEEE, 35\u201338."},{"key":"e_1_3_2_182_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-59719-5_5"},{"key":"e_1_3_2_183_2","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1007\/978-3-319-66845-1_2","volume-title":"International Conference on Integrated Formal Methods","author":"Ribeiro Pedro","year":"2017","unstructured":"Pedro Ribeiro, Alvaro Miyazawa, Wei Li, Ana Cavalcanti, and Jon Timmis. 2017. Modelling and verification of timed robotic controllers. In International Conference on Integrated Formal Methods. Springer, 18\u201333."},{"key":"e_1_3_2_184_2","first-page":"38","volume-title":"45th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)","author":"Rico Sergio","year":"2019","unstructured":"Sergio Rico, Emelie Engstr\u00f6m, and Martin H\u00f6st. 2019. A taxonomy for improving industry-academia communication in IoT vulnerability management. In 45th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE, 38\u201345."},{"key":"e_1_3_2_185_2","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2013.6696520"},{"key":"e_1_3_2_186_2","volume-title":"Smart Farming Workshop","author":"Rooker Martijn","year":"2018","unstructured":"Martijn Rooker, Pablo Horstrand, Aythami Salvador Rodriguez, Sebastian Lopez, Roberto Sarmiento, Jose Lopez, Ray Alejandro Lattarulo, Joshue Manuel Perez Rastelli, Zora Slavik, David Pereira, et\u00a0al. 2018. Towards improved validation of autonomous systems for smart farming. In Smart Farming Workshop. ISEP."},{"key":"e_1_3_2_187_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.1999.792604"},{"key":"e_1_3_2_188_2","first-page":"199","volume-title":"International Conference on Autonomous Agents and Multiagent Systems","author":"Rubin Sasha","year":"2015","unstructured":"Sasha Rubin. 2015. Parameterised verification of autonomous mobile-agents in static but unknown environments. In International Conference on Autonomous Agents and Multiagent Systems. ACM, 199\u2013208."},{"key":"e_1_3_2_189_2","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-019-00516-z"},{"key":"e_1_3_2_190_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ifacol.2017.08.1236"},{"key":"e_1_3_2_191_2","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1145\/3022099.3022102","volume-title":"3rd Workshop on Model-driven Robot Software Engineering","author":"Saglietti Francesca","year":"2016","unstructured":"Francesca Saglietti and Matthias Meitner. 2016. Model-driven structural and statistical testing of robot cooperation and reconfiguration. In 3rd Workshop on Model-driven Robot Software Engineering. ACM, 17\u201323."},{"key":"e_1_3_2_192_2","first-page":"144","volume-title":"International Conference on Computer Safety, Reliability, and Security","author":"Saglietti Francesca","year":"2014","unstructured":"Francesca Saglietti, Stefan Winzinger, and Raimar Lill. 2014. Reconfiguration testing for cooperating autonomous agents. In International Conference on Computer Safety, Reliability, and Security. Springer, 144\u2013155."},{"key":"e_1_3_2_193_2","first-page":"56","volume-title":"9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation","author":"Santos Andr\u00e9","year":"2018","unstructured":"Andr\u00e9 Santos, Alcino Cunha, and Nuno Macedo. 2018. Property-based testing for the robot operating system. In 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation. ACM, 56\u201362."},{"key":"e_1_3_2_194_2","first-page":"1","volume-title":"14th IEEE International Workshop on Factory Communication Systems (WFCS)","author":"Satoh Ichiro","year":"2018","unstructured":"Ichiro Satoh. 2018. An approach for testing software on networked transport robots. In 14th IEEE International Workshop on Factory Communication Systems (WFCS). IEEE, 1\u20134."},{"key":"e_1_3_2_195_2","doi-asserted-by":"publisher","DOI":"10.5220\/0007714503150321"},{"key":"e_1_3_2_196_2","first-page":"1083","volume-title":"Internationales Stuttgarter Symposium","author":"Sch\u00f6ner Hans-Peter","year":"2018","unstructured":"Hans-Peter Sch\u00f6ner. 2018. Simulation in development and testing of autonomous vehicles. In Internationales Stuttgarter Symposium. Springer, 1083\u20131095."},{"key":"e_1_3_2_197_2","doi-asserted-by":"crossref","first-page":"1988","DOI":"10.1109\/CCTA.2017.8062747","volume-title":"IEEE Conference on Control Technology and Applications (CCTA)","author":"Seiferth David","year":"2017","unstructured":"David Seiferth and Matthias Heller. 2017. Testing and performance enhancement of a model-based designed ground controller for a diamond-shaped unmanned air vehicle (UAV). In IEEE Conference on Control Technology and Applications (CCTA). IEEE, 1988\u20131994."},{"key":"e_1_3_2_198_2","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1007\/978-3-030-27008-7_9","volume-title":"International Workshop on Formal Methods for Industrial Critical Systems","author":"Selvaraj Yuvaraj","year":"2019","unstructured":"Yuvaraj Selvaraj, Wolfgang Ahrendt, and Martin Fabian. 2019. Verification of decision making software in an autonomous vehicle: An industrial case study. In International Workshop on Formal Methods for Industrial Critical Systems. Springer, 143\u2013159."},{"key":"e_1_3_2_199_2","doi-asserted-by":"crossref","first-page":"621","DOI":"10.1007\/978-3-319-67361-5_40","volume-title":"Field and Service Robotics","author":"Shah Shital","year":"2018","unstructured":"Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. 2018. AirSim: High-fidelity visual and physical simulation for autonomous vehicles. In Field and Service Robotics. Springer, Zurich, Switzerland, 621\u2013635."},{"key":"e_1_3_2_200_2","first-page":"1","volume-title":"35th International Conference on Computer-aided Design","author":"Shi Weijing","year":"2016","unstructured":"Weijing Shi, Mohamed Baker Alawieh, Xin Li, Huafeng Yu, Nikos Arechiga, and Nobuyuki Tomatsu. 2016. Efficient statistical validation of machine learning systems for autonomous driving. In 35th International Conference on Computer-aided Design. ACM, 1\u20138."},{"key":"e_1_3_2_201_2","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1007\/978-3-319-47443-4_12","volume-title":"IFIP International Conference on Testing Software and Systems","author":"Sippl Christoph","year":"2016","unstructured":"Christoph Sippl, Florian Bock, David Wittmann, Harald Altinger, and Reinhard German. 2016. From simulation data to test cases for fully automated driving and ADAS. In IFIP International Conference on Testing Software and Systems. Springer, 191\u2013206."},{"key":"e_1_3_2_202_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10472-011-9270-x"},{"key":"e_1_3_2_203_2","first-page":"020045","volume-title":"AIP Conference Proceedings","author":"Siwek Micha\u0142","year":"2019","unstructured":"Micha\u0142 Siwek, Leszek Baranowski, Jaros\u0142aw Panasiuk, and Wojciech Kaczmarek. 2019. Modeling and simulation of movement of dispersed group of mobile robots using Simscape multibody software. In AIP Conference Proceedings. AIP Publishing LLC, 020045."},{"key":"e_1_3_2_204_2","first-page":"81","volume-title":"44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)","author":"Spislaender Marc","year":"2018","unstructured":"Marc Spislaender and Francesca Saglietti. 2018. Evidence-based verification of safety properties concerning the cooperation of autonomous agents. In 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE, 81\u201388."},{"key":"e_1_3_2_205_2","first-page":"875","volume-title":"IEEE\/ACIS 16th International Conference on Computer and Information Science (ICIS)","author":"Sumida Tomoo","year":"2017","unstructured":"Tomoo Sumida, Hiroyuki Suzuki, Sho Sei Shun, Kazuhito Omaki, Takaaki Goto, and Kensei Tsuchida. 2017. FDR verification of a system involving a robot climbing stairs. In IEEE\/ACIS 16th International Conference on Computer and Information Science (ICIS). IEEE, 875\u2013878."},{"key":"e_1_3_2_206_2","doi-asserted-by":"publisher","DOI":"10.1145\/3302504.3311802"},{"key":"e_1_3_2_207_2","first-page":"1","volume-title":"IEEE International Conference on Connected Vehicles and Expo (ICCVE)","author":"Szalay Zsolt","year":"2019","unstructured":"Zsolt Szalay, M\u00e1ty\u00e1s Szalai, B\u00e1lint T\u00f3th, Tam\u00e1s Tettamanti, and Viktor Tihanyi. 2019. Proof of concept for Scenario-in-the-Loop (SciL) testing for autonomous vehicle technology. In IEEE International Conference on Connected Vehicles and Expo (ICCVE). IEEE, 1\u20135."},{"key":"e_1_3_2_208_2","doi-asserted-by":"publisher","DOI":"10.1109\/AITEST49225.2020.00011"},{"key":"e_1_3_2_209_2","first-page":"234","volume-title":"IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)","author":"Tao Jianbo","year":"2019","unstructured":"Jianbo Tao, Yihao Li, Franz Wotawa, Hermann Felbinger, and Mihai Nica. 2019. On the industrial application of combinatorial testing for autonomous driving functions. In IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 234\u2013240."},{"key":"e_1_3_2_210_2","volume-title":"Driving Simulation Conference (DSC)","author":"Tatar Mugur","year":"2015","unstructured":"Mugur Tatar. 2015. Enhancing ADAS test and validation with automated search for critical situations. In Driving Simulation Conference (DSC). DSC Europe."},{"key":"e_1_3_2_211_2","unstructured":"Unity Technologies. 2021. Unity. Retrieved from https:\/\/unity.com."},{"key":"e_1_3_2_212_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_3_2_213_2","first-page":"1","volume-title":"OCEANS Conference","author":"Tosik Thomas","year":"2016","unstructured":"Thomas Tosik, Jasper Schwinghammer, Mandy Jane Feldvo\u00df, John Paul Jonte, Arne Brech, and Erik Maehle. 2016. MARS: A simulation environment for marine swarm robotics and environmental monitoring. In OCEANS Conference. IEEE, 1\u20136."},{"key":"e_1_3_2_214_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-51532-8_15"},{"key":"e_1_3_2_215_2","doi-asserted-by":"crossref","first-page":"577","DOI":"10.1109\/MODELS-C.2019.00088","volume-title":"ACM\/IEEE 22nd International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C)","author":"Uriagereka Garazi Juez","year":"2019","unstructured":"Garazi Juez Uriagereka, Estibaliz Amparan, Cristina Martinez Martinez, Jabier Martinez, Aurelien Ibanez, Matteo Morelli, Ansgar Radermacher, and Huascar Espinoza. 2019. Design-time safety assessment of robotic systems using fault injection simulation in a model-driven approach. In ACM\/IEEE 22nd International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C). IEEE, 577\u2013586."},{"key":"e_1_3_2_216_2","first-page":"1","volume-title":"IEEE Aerospace Conference","author":"Verma Vandi","year":"2019","unstructured":"Vandi Verma and Chris Leger. 2019. SSim: NASA Mars Rover robotics flight software simulation. In IEEE Aerospace Conference. IEEE, 1\u201311."},{"issue":"1","key":"e_1_3_2_217_2","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1109\/TRO.2019.2937471","article-title":"Safety assessment of collaborative robotics through automated formal verification","volume":"36","author":"Vicentini Federico","year":"2019","unstructured":"Federico Vicentini, Mehrnoosh Askarpour, Matteo G. Rossi, and Dino Mandrioli. 2019. Safety assessment of collaborative robotics through automated formal verification. IEEE Trans. Robot. 36, 1 (2019), 42\u201361.","journal-title":"IEEE Trans. Robot."},{"key":"e_1_3_2_218_2","first-page":"714","volume-title":"Intelligent Systems Conference (IntelliSys)","author":"Vishnukumar Harsha Jakkanahalli","year":"2017","unstructured":"Harsha Jakkanahalli Vishnukumar, Bj\u00f6rn Butting, Christian M\u00fcller, and Eric Sax. 2017. Machine learning and deep neural network\u2013Artificial intelligence core for lab and real-world test and validation for ADAS and autonomous vehicles: AI for efficient and quality test and validation. In Intelligent Systems Conference (IntelliSys). IEEE, 714\u2013721."},{"key":"e_1_3_2_219_2","doi-asserted-by":"crossref","first-page":"347","DOI":"10.1007\/978-3-642-15651-9_26","volume-title":"International Conference on Computer Safety, Reliability, and Security","author":"Walter Dennis","year":"2010","unstructured":"Dennis Walter, Holger T\u00e4ubig, and Christoph L\u00fcth. 2010. Experiences in applying formal verification in robotics. In International Conference on Computer Safety, Reliability, and Security. Springer, 347\u2013360."},{"key":"e_1_3_2_220_2","first-page":"310","volume-title":"36th International Symposium on Automation and Robotics in Construction","author":"Wang Kai","year":"2019","unstructured":"Kai Wang and J. C. Cheng. 2019. Integrating hardware-in-the-loop simulation and BIM for planning UAV-based As-built MEP inspection with deep learning techniques. In 36th International Symposium on Automation and Robotics in Construction. IAARC, 310\u2013316."},{"key":"e_1_3_2_221_2","doi-asserted-by":"publisher","DOI":"10.1109\/TII.2017.2788901"},{"key":"e_1_3_2_222_2","doi-asserted-by":"publisher","DOI":"10.1109\/THMS.2015.2425139"},{"key":"e_1_3_2_223_2","article-title":"Formal verification of an autonomous personal robotic assistant","author":"Webster Matt","year":"2014","unstructured":"Matt Webster, Maha Salem, Clare Dixon, Michael Fisher, and Kerstin Dautenhahn. 2014. Formal verification of an autonomous personal robotic assistant. In AAAI Spring Symposium.","journal-title":"AAAI Spring Symposium"},{"issue":"1","key":"e_1_3_2_224_2","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1177\/0278364919883338","article-title":"A corroborative approach to verification and validation of human\u2013robot teams","volume":"39","author":"Webster Matt","year":"2019","unstructured":"Matt Webster, David Western, Dejanira Araiza-Illan, Clare Dixon, Kerstin Eder, Michael Fisher, and Anthony G. Pipe. 2019. A corroborative approach to verification and validation of human\u2013robot teams. Int. J. Robot. Res. 39, 1 (2019), 73\u201399.","journal-title":"Int. J. Robot. Res."},{"key":"e_1_3_2_225_2","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1109\/SIMPAR.2018.8376277","volume-title":"IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR)","author":"Wigand Dennis Leroy","year":"2018","unstructured":"Dennis Leroy Wigand, Pouya Mohammadi, Enrico Mingo Hoffman, Nikos G. Tsagarakis, Jochen J. Steil, and Sebastian Wrede. 2018. An open-source architecture for simulation, execution and analysis of real-time robotics systems. In IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR). IEEE, 93\u2013100."},{"key":"e_1_3_2_226_2","doi-asserted-by":"publisher","DOI":"10.1145\/2331147.2331163"},{"key":"e_1_3_2_227_2","first-page":"128","volume-title":"International Symposium on Theoretical Aspects of Software Engineering (TASE)","author":"Xu Bingqing","year":"2019","unstructured":"Bingqing Xu, Qin Li, Tong Guo, Yi Ao, and Dehui Du. 2019. A quantitative safety verification approach for the decision-making process of autonomous driving. In International Symposium on Theoretical Aspects of Software Engineering (TASE). IEEE, 128\u2013135."},{"key":"e_1_3_2_228_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2019.2943184"},{"issue":"11","key":"e_1_3_2_229_2","first-page":"13555","article-title":"Behavioral modeling and verification of multi-agent systems for manufacturing control","volume":"38","author":"Yeung Wing Lok","year":"2011","unstructured":"Wing Lok Yeung. 2011. Behavioral modeling and verification of multi-agent systems for manufacturing control. Exp. Syst. applic. 38, 11 (2011), 13555\u201313562.","journal-title":"Exp. Syst. applic."},{"key":"e_1_3_2_230_2","first-page":"1","volume-title":"Symposium on Modeling and Simulation of Complexity in Intelligent, Adaptive and Autonomous Systems","author":"Yilmaz Levent","year":"2017","unstructured":"Levent Yilmaz. 2017. Verification and validation of ethical decision-making in autonomous systems. In Symposium on Modeling and Simulation of Complexity in Intelligent, Adaptive and Autonomous Systems. Springer, 1\u201312."},{"key":"e_1_3_2_231_2","first-page":"14","volume-title":"International Conference on Embedded Systems and Applications (ESA)","author":"Yujian Fu","year":"2014","unstructured":"Fu Yujian and Drabo Mebougna. 2014. Formal modeling and verification of dynamic reconfiguration of autonomous robotics systems. In International Conference on Embedded Systems and Applications (ESA). Csrea, 14."},{"issue":"5","key":"e_1_3_2_232_2","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1109\/MCE.2019.2923899","article-title":"Human\u2013machine interface design and verification for an automated driving system using system model and driving simulator","volume":"8","author":"Yun Sunkil","year":"2019","unstructured":"Sunkil Yun, Takaaki Teshima, and Hidekazu Nishimura. 2019. Human\u2013machine interface design and verification for an automated driving system using system model and driving simulator. IEEE Consum. Electron. Mag. 8, 5 (2019), 92\u201398.","journal-title":"IEEE Consum. Electron. Mag."},{"key":"e_1_3_2_233_2","doi-asserted-by":"crossref","first-page":"1160","DOI":"10.1109\/ITSC.2014.6957844","volume-title":"17th International IEEE Conference on Intelligent Transportation Systems (ITSC)","author":"Zhang Chi","year":"2014","unstructured":"Chi Zhang, Yuehu Liu, Danchen Zhao, and Yuanqi Su. 2014. RoadView: A traffic scene simulator for autonomous vehicle simulation testing. In 17th International IEEE Conference on Intelligent Transportation Systems (ITSC). IEEE, 1160\u20131165."},{"key":"e_1_3_2_234_2","doi-asserted-by":"crossref","first-page":"858","DOI":"10.1109\/ROBIO49542.2019.8961433","volume-title":"IEEE International Conference on Robotics and Biomimetics (ROBIO)","author":"Zhang Xiaoyang","year":"2019","unstructured":"Xiaoyang Zhang, Hongpeng Wang, Jingtai Liu, and Haifeng Li. 2019. CyberEarth: A virtual simulation platform for robotics and cyber-physical systems. In IEEE International Conference on Robotics and Biomimetics (ROBIO). IEEE, 858\u2013863."},{"key":"e_1_3_2_235_2","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1007\/978-3-030-30446-1_6","volume-title":"International Conference on Software Engineering and Formal Methods","author":"Zhao Xingyu","year":"2019","unstructured":"Xingyu Zhao, Matt Osborne, Jenny Lantair, Valentin Robu, David Flynn, Xiaowei Huang, Michael Fisher, Fabio Papacchini, and Angelo Ferrando. 2019. Towards integrating formal verification of autonomous robots with battery prognostics and health management. In International Conference on Software Engineering and Formal Methods. Springer, 105\u2013124."},{"key":"e_1_3_2_236_2","first-page":"8066","volume-title":"AAAI Conference on Artificial Intelligence","author":"Zhao Xingyu","year":"2019","unstructured":"Xingyu Zhao, Valentin Robu, David Flynn, Fateme Dinmohammadi, Michael Fisher, and Matt Webster. 2019. Probabilistic model checking of robots deployed in extreme environments. In AAAI Conference on Artificial Intelligence. AAAI, 8066\u20138074."},{"key":"e_1_3_2_237_2","first-page":"13","volume-title":"IEEE 30th International Symposium on Software Reliability Engineering (ISSRE)","author":"Zhao Xingyu","year":"2019","unstructured":"Xingyu Zhao, Valentin Robu, David Flynn, Kizito Salako, and Lorenzo Strigini. 2019. Assessing the safety and reliability of autonomous vehicles from road testing. In IEEE 30th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 13\u201323."},{"issue":"2016","key":"e_1_3_2_238_2","doi-asserted-by":"crossref","first-page":"66","DOI":"10.4271\/2016-01-0049","article-title":"A framework for virtual testing of ADAs","volume":"9","author":"Zhou Jinwei","year":"2016","unstructured":"Jinwei Zhou, Roman Schmied, Alexander Sandalek, Helmut Kokal, and Luigi del Re. 2016. A framework for virtual testing of ADAs. SAE Int. J. Passeng. Cars-Electron. Electric. Syst. 9, 2016-01-0049 (2016), 66\u201373.","journal-title":"SAE Int. J. Passeng. Cars-Electron. Electric. Syst."},{"key":"e_1_3_2_239_2","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1109\/SIMPAR.2018.8376285","volume-title":"IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR)","author":"Zofka Marc Ren\u00e9","year":"2018","unstructured":"Marc Ren\u00e9 Zofka, Marc Essinger, Tobias Fleck, Ralf Kohlhaas, and J. Marius Z\u00f6llner. 2018. The sleepwalker framework: Verification and validation of autonomous vehicles by mixed reality lidar stimulation. In IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR). IEEE, 151\u2013157."}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3542945","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3542945","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T13:49:31Z","timestamp":1750168171000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3542945"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,30]]},"references-count":238,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2023,4,30]]}},"alternative-id":["10.1145\/3542945"],"URL":"https:\/\/doi.org\/10.1145\/3542945","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,30]]},"assertion":[{"value":"2021-01-27","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-05-13","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-03-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}