{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,19]],"date-time":"2026-05-19T14:09:55Z","timestamp":1779199795206,"version":"3.51.4"},"reference-count":108,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2024,3,15]],"date-time":"2024-03-15T00:00:00Z","timestamp":1710460800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"European Union\u2019s Horizon 2020 and Horizon Europe","award":["956123"],"award-info":[{"award-number":["956123"]}]},{"name":"NWO","award":["14897 P14-18"],"award-info":[{"award-number":["14897 P14-18"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,3,31]]},"abstract":"<jats:p>The automotive industry is now known for its software-intensive and safety-critical nature. The industry is on a path to the holy grail of completely automating driving, starting from relatively simple operational areas like highways. One of the most challenging, evolving, and essential parts of automated driving is the software that enables understanding of surroundings and the vehicle\u2019s own as well as surrounding objects\u2019 relative position, otherwise known as the perception system. Current generation perception systems are formed by a combination of traditional software and machine learning-related software. With automated driving systems transitioning from research to production, it is imperative to assess their safety.<\/jats:p>\n          <jats:p\/>\n          <jats:p>We assess the safety of Apollo, the most popular open-source automotive software, at the design level for its use on a Dutch highway. We identified 58 safety requirements, 38 of which are found to be fulfilled at the design level. We observe that all requirements relating to traditional software are fulfilled, while most requirements specific to machine learning systems are not. This study unveils issues that need immediate attention; and directions for future research to make automated driving safe.<\/jats:p>","DOI":"10.1145\/3631969","type":"journal-article","created":{"date-parts":[[2023,11,8]],"date-time":"2023-11-08T11:56:55Z","timestamp":1699444615000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":18,"title":["Safety of Perception Systems for Automated Driving: A Case Study on Apollo"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1748-3147","authenticated-orcid":false,"given":"Sangeeth","family":"Kochanthara","sequence":"first","affiliation":[{"name":"Eindhoven University of Technology, Eindhoven, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-6508-7264","authenticated-orcid":false,"given":"Tajinder","family":"Singh","sequence":"additional","affiliation":[{"name":"Siemens Digital Industries Software, Helmond, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0207-1470","authenticated-orcid":false,"given":"Alexandru","family":"Forrai","sequence":"additional","affiliation":[{"name":"Siemens Digital Industries Software, Helmond, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7221-3676","authenticated-orcid":false,"given":"Loek","family":"Cleophas","sequence":"additional","affiliation":[{"name":"Eindhoven University of Technology, Eindhoven, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,3,15]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Apollo automated driving platform. Retrieved from https:\/\/github.com\/ApolloAuto\/apollo"},{"key":"e_1_3_2_3_2","unstructured":"ISO\/PAS 21448:2019 Road vehicles \u2013 Safety of the intended functionality. Retrieved from https:\/\/www.iso.org\/standard\/70939.html"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","unstructured":"Replication Package: Safety of Perception Systems for Automated Driving: A Case Study on Apollo. DOI:10.5281\/zenodo.8226271","DOI":"10.5281\/zenodo.8226271"},{"key":"e_1_3_2_5_2","unstructured":"Road design - Which road categories are distinguished in the Netherlands? Retrieved from https:\/\/www.swov.nl\/en\/facts-figures\/fact\/road-design-which-road-categories-are-distinguished-netherlands"},{"key":"e_1_3_2_6_2","unstructured":"Road traffic signs and regulations in the Netherlands. Retrieved from https:\/\/www.universiteitleiden.nl\/binaries\/content\/assets\/customsites\/study-abroad-exchange-students\/road_traffic_signs_and_regulations_jan_2013_uk.pdf"},{"key":"e_1_3_2_7_2","unstructured":"Safety first for automated driving 2019. Retrieved from https:\/\/www.daimler.com\/documents\/innovation\/other\/safety-first-for-automated-driving.pdf"},{"key":"e_1_3_2_8_2","unstructured":"Self-driving safety report. Retrieved from https:\/\/www.gm.com\/content\/dam\/company\/docs\/us\/en\/gmcom\/gmsafetyreport.pdf"},{"key":"e_1_3_2_9_2","unstructured":"Waymo Safety Report: on the Road to Fully Self-driving. Retrieved from https:\/\/assets.ctfassets.net\/sv23gofxcuiz\/1xAGjnH0kTxD2Vmvqsv3eS\/7da216660cbd9ee7b35eefcac1b28a2f\/waymo-safety-report-2018.pdf"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3450356"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1155\/2019\/5015741"},{"key":"e_1_3_2_12_2","article-title":"Concrete problems in AI safety","author":"Amodei Dario","year":"2016","unstructured":"Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man\u00e9. 2016. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565 (2016).","journal-title":"arXiv preprint arXiv:1606.06565"},{"key":"e_1_3_2_13_2","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1145\/3194085.3194091","volume-title":"IEEE\/ACM 1st International Workshop on Software Engineering for AI in Autonomous Systems (SEFAIAS\u201918)","author":"Aniculaesei Adina","year":"2018","unstructured":"Adina Aniculaesei, J\u00f6rg Grieser, Andreas Rausch, Karina Rehfeldt, and Tim Warnecke. 2018. Toward a holistic software systems engineering approach for dependable autonomous systems. In IEEE\/ACM 1st International Workshop on Software Engineering for AI in Autonomous Systems (SEFAIAS\u201918). IEEE, 23\u201330."},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3417038"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/IROS51168.2021.9636497"},{"key":"e_1_3_2_16_2","volume-title":"AVSC Best Practice for Describing an Operational Design Domain: Conceptual Framework and Lexicon","year":"2020","unstructured":"AVSC. 2020. AVSC Best Practice for Describing an Operational Design Domain: Conceptual Framework and Lexicon. Technical Report. SAE Industry Technologies Consortia."},{"key":"e_1_3_2_17_2","first-page":"691","volume-title":"IEEE Intelligent Vehicles Symposium (IV\u201916)","author":"Bagschik Gerrit","year":"2016","unstructured":"Gerrit Bagschik, Andreas Reschka, Torben Stolte, and Markus Maurer. 2016. Identification of potential hazardous events for an unmanned protective vehicle. In IEEE Intelligent Vehicles Symposium (IV\u201916). IEEE, 691\u2013697."},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.5555\/2392670"},{"key":"e_1_3_2_19_2","first-page":"51","volume-title":"USENIX Conference on Operational Machine Learning (OpML\u201919)","author":"Baylor Denis","year":"2019","unstructured":"Denis Baylor, Kevin Haas, Konstantinos Katsiapis, Sammy Leong, Rose Liu, Clemens Menwald, Hui Miao, Neoklis Polyzotis, Mitchell Trott, and Martin Zinkevich. 2019. Continuous training for production ML in the TensorFlow extended (TFX) Platform. In USENIX Conference on Operational Machine Learning (OpML\u201919). 51\u201353."},{"key":"e_1_3_2_20_2","volume-title":"Safety of the Intended Functionality of Lane-centering and Lane-changing Maneuvers of a Generic Level 3 Highway Chauffeur System","author":"Becker Christopher","year":"2020","unstructured":"Christopher Becker, John C. Brewer, and Larry Yount. 2020. Safety of the Intended Functionality of Lane-centering and Lane-changing Maneuvers of a Generic Level 3 Highway Chauffeur System. Technical Report. US DOT National Highway Traffic Safety."},{"key":"e_1_3_2_21_2","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1145\/2752489.2752491","volume-title":"1st International Workshop on Automotive Software Architecture (WASA\u201915)","author":"Behere Sagar","year":"2015","unstructured":"Sagar Behere and Martin Torngren. 2015. A functional architecture for autonomous driving. In 1st International Workshop on Automotive Software Architecture (WASA\u201915). IEEE, 3\u201310."},{"key":"e_1_3_2_22_2","first-page":"760","volume-title":"IEEE Intelligent Vehicles Symposium (IV\u201918)","author":"Bijelic Mario","year":"2018","unstructured":"Mario Bijelic, Tobias Gruber, and Werner Ritter. 2018. A benchmark for LiDAR sensors in fog: Is detection breaking down? In IEEE Intelligent Vehicles Symposium (IV\u201918). IEEE, 760\u2013767."},{"key":"e_1_3_2_23_2","article-title":"Preliminary report, highway, HWY18mh010","author":"Board National Transportation Safety","year":"2018","unstructured":"National Transportation Safety Board. 2018. Preliminary report, highway, HWY18mh010. Preliminary Report on First Autonomous Vehicle Test Car Fatality in Arizona on (2018). https:\/\/www.ntsb.gov\/investigations\/Pages\/HWY18MH010.aspx","journal-title":"Preliminary Report on First Autonomous Vehicle Test Car Fatality in Arizona on"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/BigData.2017.8258038"},{"key":"e_1_3_2_25_2","first-page":"1094","volume-title":"IEEE Intelligent Vehicles Symposium (IV\u201920)","author":"Carballo Alexander","year":"2020","unstructured":"Alexander Carballo, Jacob Lambert, Abraham Monrroy, David Wong, Patiphon Narksri, Yuki Kitsukawa, Eijiro Takeuchi, Shinpei Kato, and Kazuya Takeda. 2020. LIBRE: The multiple 3D LiDAR dataset. In IEEE Intelligent Vehicles Symposium (IV\u201920). IEEE, 1094\u20131101."},{"key":"e_1_3_2_26_2","article-title":"MMDetection: Open MMLab detection toolbox and benchmark","author":"Chen Kai","year":"2019","unstructured":"Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. 2019. MMDetection: Open MMLab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155 (2019).","journal-title":"arXiv preprint arXiv:1906.07155"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1177\/001316446002000104"},{"key":"e_1_3_2_28_2","unstructured":"MMDetection3D Contributors. 2020. MMDetection3D: OpenMMLab next-generation platform for general 3D object detection. Retrieved on 08-August-2023 https:\/\/github.com\/open-mmlab\/mmdetection3d"},{"key":"e_1_3_2_29_2","article-title":"Automated Driving System (ADS) High\u2013level Quality Requirements Analysis\u2013Driving Behavior Safety","author":"Czarnecki Krzysztof","year":"2018","unstructured":"Krzysztof Czarnecki. 2018. Automated Driving System (ADS) High\u2013level Quality Requirements Analysis\u2013Driving Behavior Safety. Waterloo Intelligent Systems Engineering Lab (WISE) Report, University of Waterloo.","journal-title":"Waterloo Intelligent Systems Engineering Lab (WISE) Report, University of Waterloo"},{"key":"e_1_3_2_30_2","first-page":"1050","volume-title":"International Conference on Machine Learning","author":"Gal Yarin","year":"2016","unstructured":"Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning. PMLR, 1050\u20131059."},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338942"},{"key":"e_1_3_2_32_2","first-page":"1180","volume-title":"International Conference on Machine Learning","author":"Ganin Yaroslav","year":"2015","unstructured":"Yaroslav Ganin and Victor Lempitsky. 2015. Unsupervised domain adaptation by backpropagation. In International Conference on Machine Learning. PMLR, 1180\u20131189."},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409743"},{"key":"e_1_3_2_34_2","article-title":"Selective classification for deep neural networks","volume":"30","author":"Geifman Yonatan","year":"2017","unstructured":"Yonatan Geifman and Ran El-Yaniv. 2017. Selective classification for deep neural networks. Adv. Neural Inf. Process. Syst. 30 (2017).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_2_35_2","first-page":"2151","volume-title":"International Conference on Machine Learning","author":"Geifman Yonatan","year":"2019","unstructured":"Yonatan Geifman and Ran El-Yaniv. 2019. SelectiveNet: A deep neural network with an integrated reject option. In International Conference on Machine Learning. PMLR, 2151\u20132159."},{"key":"e_1_3_2_36_2","article-title":"ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness","author":"Geirhos Robert","year":"2018","unstructured":"Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. 2018. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv:1811.12231 (2018).","journal-title":"arXiv preprint arXiv:1811.12231"},{"key":"e_1_3_2_37_2","article-title":"Deep anomaly detection using geometric transformations","volume":"31","author":"Golan Izhak","year":"2018","unstructured":"Izhak Golan and Ran El-Yaniv. 2018. Deep anomaly detection using geometric transformations. Adv. Neural Inf. Process. Syst. 31 (2018).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_2_38_2","first-page":"1321","volume-title":"International Conference on Machine Learning","author":"Guo Chuan","year":"2017","unstructured":"Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In International Conference on Machine Learning. PMLR, 1321\u20131330."},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1049\/iet-sen.2013.0236"},{"key":"e_1_3_2_40_2","first-page":"2712","volume-title":"International Conference on Machine Learning","author":"Hendrycks Dan","year":"2019","unstructured":"Dan Hendrycks, Kimin Lee, and Mantas Mazeika. 2019. Using pre-training can improve model robustness and uncertainty. In International Conference on Machine Learning. PMLR, 2712\u20132721."},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.2307\/25148625"},{"key":"e_1_3_2_42_2","volume-title":"AI Engineering: 11 Foundational Practices","author":"Horneman Angela","year":"2020","unstructured":"Angela Horneman, Andrew Mellinger, and Ipek Ozkaya. 2020. AI Engineering: 11 Foundational Practices. Technical Report. Carnegie Mellon University Pittsburgh, PA."},{"key":"e_1_3_2_43_2","article-title":"Multi-scale dense networks for resource efficient image classification","author":"Huang Gao","year":"2017","unstructured":"Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens Van Der Maaten, and Kilian Q. Weinberger. 2017. Multi-scale dense networks for resource efficient image classification. arXiv preprint arXiv:1703.09844 (2017).","journal-title":"arXiv preprint arXiv:1703.09844"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2020.2977911"},{"key":"e_1_3_2_45_2","volume-title":"IEC Functional Safety and IEC 61508","year":"2010","unstructured":"IEC. 2010. IEC Functional Safety and IEC 61508. Standard. International Electrotechnical Commission."},{"key":"e_1_3_2_46_2","volume-title":"ISO 26262: 2018 - Road Vehicles \u2013 Functional Safety","year":"2018","unstructured":"ISO. 2018. ISO 26262: 2018 - Road Vehicles \u2013 Functional Safety. Standard. International Organization for Standardization."},{"key":"e_1_3_2_47_2","volume-title":"ISO\/PAS 21448: 2019 - Road Vehicles \u2013 Safety of the Intended Functionality","year":"2019","unstructured":"ISO. 2019. ISO\/PAS 21448: 2019 - Road Vehicles \u2013 Safety of the Intended Functionality. Standard. International Organization for Standardization."},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.5555\/2025240"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3417065"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0951-8320(96)00100-7"},{"key":"e_1_3_2_51_2","volume-title":"REVERT: RuntimE VEerification for Real-Time Systems","author":"Kochanthara Sangeeth","year":"2016","unstructured":"Sangeeth Kochanthara. 2016. REVERT: RuntimE VEerification for Real-Time Systems. Master\u2019s Thesis. Indraprastha Institute of Information Technology Delhi, India. Retrieved from https:\/\/repository.iiitd.edu.in\/xmlui\/bitstream\/handle\/123456789\/524\/MT14055%20-%20Sangeeth%20K.pdf"},{"key":"e_1_3_2_52_2","volume-title":"A Changing Landscape: On Safety & Open Source in Automated and Connected Driving","author":"Kochanthara Sangeeth","year":"2023","unstructured":"Sangeeth Kochanthara. 2023. A Changing Landscape: On Safety & Open Source in Automated and Connected Driving. Ph.D. Dissertation. Eindhoven University of Technology, The Netherlands. Retrieved from https:\/\/research.tue.nl\/en\/publications\/a-changing-landscape-on-safety-amp-open-source-in-automated-and-c"},{"key":"e_1_3_2_53_2","volume-title":"Mining Software Repositories Conference","author":"Kochanthara Sangeeth","unstructured":"Sangeeth Kochanthara, Yanja Dajsuren, Loek Cleophas, and Mark van den Brand. [n.d.]. Painting the landscape of automotive software in GitHub. In Mining Software Repositories Conference. Retrieved from https:\/\/arxiv.org\/pdf\/2203.08936.pdf"},{"key":"e_1_3_2_54_2","first-page":"365","volume-title":"IEEE Real-Time Systems Symposium","author":"Kochanthara Sangeeth","year":"2016","unstructured":"Sangeeth Kochanthara, Geoffrey Nelissen, David Pereira, and Rahul Purandare. 2016. REVERT: A monitor generation tool for real-time systems. In IEEE Real-Time Systems Symposium. 365\u2013365."},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSA-C50368.2020.00018"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2021.110991"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.trf.2015.04.014"},{"key":"e_1_3_2_58_2","article-title":"Simple and scalable predictive uncertainty estimation using deep ensembles","volume":"30","author":"Lakshminarayanan Balaji","year":"2017","unstructured":"Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. Adv. Neural Inf. Process. Syst. 30 (2017).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.01298"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2019.2938879"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/TR.1985.5222114"},{"key":"e_1_3_2_62_2","article-title":"AI safety gridworlds","author":"Leike Jan","year":"2017","unstructured":"Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A. Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, and Shane Legg. 2017. AI safety gridworlds. arXiv preprint arXiv:1711.09883 (2017).","journal-title":"arXiv preprint arXiv:1711.09883"},{"key":"e_1_3_2_63_2","first-page":"1","article-title":"STPA handbook","volume":"3","author":"Leveson Nancy G.","year":"2018","unstructured":"Nancy G. Leveson and John P. Thomas. 2018. STPA handbook. NANCY LEVESON AND JOHN THOMAS 3 (2018), 1\u2013188. https:\/\/psas.scripts.mit.edu\/home\/get_file.php?name=STPA_handbook.pdf. Retrieved on: August 2022.","journal-title":"NANCY LEVESON AND JOHN THOMAS"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2020.2998077"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1561\/2400000035"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW50498.2020.00506"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2016.04.002"},{"key":"e_1_3_2_68_2","unstructured":"V. M. Megler. 2019. Managing Machine Learning Projects. (2019). https:\/\/d1.awsstatic.com\/whitepapers\/aws-managing-ml-projects.pdf"},{"key":"e_1_3_2_69_2","article-title":"Benchmarking robustness in object detection: Autonomous driving when winter is coming","author":"Michaelis Claudio","year":"2019","unstructured":"Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexander S. Ecker, Matthias Bethge, and Wieland Brendel. 2019. Benchmarking robustness in object detection: Autonomous driving when winter is coming. arXiv preprint arXiv:1907.07484 (2019).","journal-title":"arXiv preprint arXiv:1907.07484"},{"key":"e_1_3_2_70_2","article-title":"Predicting model failure using saliency maps in autonomous driving systems","author":"Mohseni Sina","year":"2019","unstructured":"Sina Mohseni, Akshay Jagadeesh, and Zhangyang Wang. 2019. Predicting model failure using saliency maps in autonomous driving systems. arXiv preprint arXiv:1905.07679 (2019).","journal-title":"arXiv preprint arXiv:1905.07679"},{"key":"e_1_3_2_71_2","first-page":"162","article-title":"Practical solutions for machine learning safety in autonomous vehicles","author":"Mohseni Sina","year":"2020","unstructured":"Sina Mohseni, Mandar Pitale, Vasu Singh, and Zhangyang Wang. 2020. Practical solutions for machine learning safety in autonomous vehicles. In Workshop on Artificial Intelligence Safety, co-located with 34th AAAI Conference on Artificial Intelligence (SafeAI@AAAI\u201920). 162\u2013169. Retrieved from https:\/\/ceur-ws.org\/Vol-2560\/paper40.pdf","journal-title":"Workshop on Artificial Intelligence Safety, co-located with 34th AAAI Conference on Artificial Intelligence (SafeAI@AAAI\u201920)"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i04.5966"},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1145\/3282517.3302397"},{"key":"e_1_3_2_74_2","article-title":"Commission recommendation of 22 December 2006 on safe and efficient in-vehicle information and communication systems: Update of the european statement of principles on human machine interface","author":"Communities. Commission of the European","year":"2006","unstructured":"Commission of the European Communities.2006. Commission recommendation of 22 December 2006 on safe and efficient in-vehicle information and communication systems: Update of the european statement of principles on human machine interface. Official Update of the European Union. 78. EC (2006). Retrieved from http:\/\/data.europa.eu\/eli\/reco\/2007\/78\/oj","journal-title":"Official Update of the European Union. 78. EC"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1109\/COMPSAC.2008.139"},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.3020266"},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.2753\/MIS0742-1222240302"},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3417063"},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3054782"},{"key":"e_1_3_2_80_2","first-page":"1","volume-title":"18th European Conference on Pattern Languages of Program","author":"Preschern Christopher","year":"2015","unstructured":"Christopher Preschern, Nermin Kajtazovic, and Christian Kreiner. 2015. Building a safety architecture pattern system. In 18th European Conference on Pattern Languages of Program. 1\u201355."},{"key":"e_1_3_2_81_2","unstructured":"Christopher Preschern Nermin Kajtazovic and Christian Kreiner. 2015. Building a safety architecture pattern system. In European Conference on Patterns and Pattern Languages(EuroPLoP\u201913). ACM 55 pages."},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1109\/WACVW52041.2021.00021"},{"key":"e_1_3_2_83_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409730"},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.1109\/SYSCON.2018.8369598"},{"key":"e_1_3_2_85_2","article-title":"Waymo public road safety performance data","author":"Schwall Matthew","year":"2020","unstructured":"Matthew Schwall, Tom Daniel, Trent Victor, Francesca Favaro, and Henning Hohnhold. 2020. Waymo public road safety performance data. arXiv preprint arXiv:2011.00038 (2020).","journal-title":"arXiv preprint arXiv:2011.00038"},{"key":"e_1_3_2_86_2","article-title":"Hidden technical debt in machine learning systems","volume":"28","author":"Sculley David","year":"2015","unstructured":"David Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. 2015. Hidden technical debt in machine learning systems. Adv. Neural Inf. Process. Syst. 28 (2015).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_2_87_2","doi-asserted-by":"publisher","DOI":"10.1145\/3398394"},{"key":"e_1_3_2_88_2","first-page":"1","volume-title":"14th ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM\u201920)","author":"Serban Alex","year":"2020","unstructured":"Alex Serban, Koen van der Blom, Holger Hoos, and Joost Visser. 2020. Adoption and effects of software engineering best practices in machine learning. In 14th ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM\u201920). 1\u201312."},{"key":"e_1_3_2_89_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER53432.2022.00029"},{"key":"e_1_3_2_90_2","volume-title":"International Symposium on Automated Technology for Verification and Analysis","author":"Seshia Sanjit A.","year":"2018","unstructured":"Sanjit A. Seshia, Ankush Desai, Tommaso Dreossi, Daniel J. Fremont, Shromona Ghosh, Edward Kim, Sumukh Shivakumar, Marcell Vazquez-Chanlatte, and Xiangyu Yue. 2018. Formal specification for deep neural networks. In International Symposium on Automated Technology for Verification and Analysis. Springer."},{"key":"e_1_3_2_91_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58568-6_45"},{"key":"e_1_3_2_92_2","volume-title":"Failure Mode and Effect Analysis: FMEA from Theory to Execution","author":"Stamatis Diomidis H.","year":"2003","unstructured":"Diomidis H. Stamatis. 2003. Failure Mode and Effect Analysis: FMEA from Theory to Execution. Quality Press."},{"key":"e_1_3_2_93_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00252"},{"key":"e_1_3_2_94_2","doi-asserted-by":"publisher","DOI":"10.1177\/0361198120901681"},{"key":"e_1_3_2_95_2","volume-title":"A Framework for Automated Driving System Testable Cases and Scenarios","year":"2018","unstructured":"Eric Thorn, Shawn Kimmel, and Michelle Chaka. 2018. A Framework for Automated Driving System Testable Cases and Scenarios. Technical Report. US DOT National Highway Traffic Safety."},{"key":"e_1_3_2_96_2","article-title":"A review of formal methods applied to machine learning","author":"Urban Caterina","year":"2021","unstructured":"Caterina Urban and Antoine Min\u00e9. 2021. A review of formal methods applied to machine learning. arXiv preprint arXiv:2104.02466 (2021).","journal-title":"arXiv preprint arXiv:2104.02466"},{"key":"e_1_3_2_97_2","doi-asserted-by":"publisher","DOI":"10.1109\/ITSC.2019.8917269"},{"key":"e_1_3_2_98_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01237-3_34"},{"key":"e_1_3_2_99_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2018.8461224"},{"key":"e_1_3_2_100_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.683"},{"key":"e_1_3_2_101_2","first-page":"49","volume-title":"10th International Workshop on Empirical Software Engineering in Practice (IWESEP\u201919)","author":"Washizaki Hironori","year":"2019","unstructured":"Hironori Washizaki, Hiromu Uchida, Foutse Khomh, and Yann-Ga\u00ebl Gu\u00e9h\u00e9neuc. 2019. Studying software engineering patterns for designing machine learning systems. In 10th International Workshop on Empirical Software Engineering in Practice (IWESEP\u201919). IEEE, 49\u2013495."},{"key":"e_1_3_2_102_2","volume-title":"SAFECOMP 2013-Workshop DECS (ERCIM\/EWICS Workshop on Dependable Embedded and Cyber-physical Systems) of the 32nd International Conference on Computer Safety, Reliability and Security","author":"Westman Jonas","year":"2013","unstructured":"Jonas Westman and Mattias Nyberg. 2013. A reference example on the specification of safety requirements using ISO 26262. In SAFECOMP 2013-Workshop DECS (ERCIM\/EWICS Workshop on Dependable Embedded and Cyber-physical Systems) of the 32nd International Conference on Computer Safety, Reliability and Security."},{"key":"e_1_3_2_103_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-662-43839-8"},{"key":"e_1_3_2_104_2","first-page":"368","volume-title":"28th Annual International Computer Software and Applications Conference (COMPSAC\u201904).","author":"Wu Weihang","year":"2004","unstructured":"Weihang Wu and Tim Kelly. 2004. Safety tactics for software architecture design. In 28th Annual International Computer Software and Applications Conference (COMPSAC\u201904). IEEE, 368\u2013375."},{"key":"e_1_3_2_105_2","doi-asserted-by":"publisher","DOI":"10.1109\/LWC.2020.2984620"},{"key":"e_1_3_2_106_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.iatssr.2019.11.005"},{"key":"e_1_3_2_107_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.2983149"},{"key":"e_1_3_2_108_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCSVT.2020.2967419"},{"key":"e_1_3_2_109_2","article-title":"Autonomous driving in adverse weather conditions: A survey","author":"Zhang Yuxiao","year":"2021","unstructured":"Yuxiao Zhang, Alexander Carballo, Hanting Yang, and Kazuya Takeda. 2021. Autonomous driving in adverse weather conditions: A survey. arXiv:2112.08936 (2021).","journal-title":"arXiv:2112.08936"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3631969","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3631969","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:51:02Z","timestamp":1750287062000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3631969"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,15]]},"references-count":108,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2024,3,31]]}},"alternative-id":["10.1145\/3631969"],"URL":"https:\/\/doi.org\/10.1145\/3631969","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,15]]},"assertion":[{"value":"2022-09-23","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-10-19","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-03-15","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}