{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,16]],"date-time":"2026-04-16T14:55:07Z","timestamp":1776351307725,"version":"3.51.2"},"reference-count":46,"publisher":"MDPI AG","issue":"13","license":[{"start":{"date-parts":[[2020,7,1]],"date-time":"2020-07-01T00:00:00Z","timestamp":1593561600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>For a safe market launch of automated vehicles, the risks of the overall system as well as the sub-components must be efficiently identified and evaluated. This also includes camera-based object detection using artificial intelligence algorithms. It is trivial and explainable that due to the principle of the camera, performance depends highly on the environmental conditions and can be poor, for example in heavy fog. However, there are other factors influencing the performance of camera-based object detection, which will be comprehensively investigated for the first time in this paper. Furthermore, a precise modeling of the detection performance and the explanation of individual detection results is not possible due to the artificial intelligence based algorithms used. Therefore, a modeling approach based on the investigated influence factors is proposed and the newly developed SHapley Additive exPlanations (SHAP) approach is adopted to analyze and explain the detection performance of different object detection algorithms. The results show that many influence factors such as the relative rotation of an object towards the camera or the position of an object on the image have basically the same influence on the detection performance regardless of the detection algorithm used. In particular, the revealed weaknesses of the tested object detectors can be used to derive challenging and critical scenarios for the testing and type approval of automated vehicles.<\/jats:p>","DOI":"10.3390\/s20133699","type":"journal-article","created":{"date-parts":[[2020,7,2]],"date-time":"2020-07-02T02:44:25Z","timestamp":1593657865000},"page":"3699","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":39,"title":["Identification and Explanation of Challenging Conditions for Camera-Based Object Detection of Automated Vehicles"],"prefix":"10.3390","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8067-1872","authenticated-orcid":false,"given":"Thomas","family":"Ponn","sequence":"first","affiliation":[{"name":"Institute of Automotive Technology, Technical University of Munich, 85748 Garching, Germany"}]},{"given":"Thomas","family":"Kr\u00f6ger","sequence":"additional","affiliation":[{"name":"Institute of Automotive Technology, Technical University of Munich, 85748 Garching, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1441-5226","authenticated-orcid":false,"given":"Frank","family":"Diermeyer","sequence":"additional","affiliation":[{"name":"Institute of Automotive Technology, Technical University of Munich, 85748 Garching, Germany"}]}],"member":"1968","published-online":{"date-parts":[[2020,7,1]]},"reference":[{"key":"ref_1","unstructured":"SAE J3016 (2020, June 30). Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles. Available online: https:\/\/www.sae.org\/standards\/content\/j3016_201806\/."},{"key":"ref_2","unstructured":"Lundberg, S.M., and Lee, S.I. (2017, January 4\u20139). A Unified Approach to Interpreting Model Predictions. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA."},{"key":"ref_3","unstructured":"Lundberg, S. (2020, April 13). SHAP (SHapley Additive exPlanations): Explainers. Available online: https:\/\/shap.readthedocs.io\/en\/latest\/."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1038\/s42256-019-0138-9","article-title":"From local explanations to global understanding with explainable AI for trees","volume":"2","author":"Lundberg","year":"2020","journal-title":"Nat. Mach. Intell."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"98","DOI":"10.1007\/s11263-014-0733-5","article-title":"The PASCAL Visual Object Classes Challenge: A Retrospective","volume":"111","author":"Everingham","year":"2015","journal-title":"Int. J. Comput. Vis."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Rebala, G., Ravi, A., and Churiwala, S. (2019). An Introduction to Machine Learning, Springer.","DOI":"10.1007\/978-3-030-15729-6"},{"key":"ref_7","unstructured":"Manning, C.D., Raghavan, P., and Schuetze, H. (2009). An Introduction to Information Retrieval, Cambridge University Press."},{"key":"ref_8","unstructured":"Zou, Z., Shi, Z., Guo, Y., and Ye, J. (2019). Object Detection in 20 Years: A Survey. arXiv."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Huang, J., Rathod, V., Sun, C., Zhu, M., and Korattikara, A. (2017, January 22\u201325). Speed\/Accuracy trade-offs for modern convolutional object detectors. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.351"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"318","DOI":"10.1109\/TPAMI.2018.2858826","article-title":"Focal Loss for Dense Object Detection","volume":"42","author":"Lin","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1007\/978-3-319-46448-0_2","article-title":"SSD: Single Shot MultiBox Detector","volume":"9905","author":"Liu","year":"2016","journal-title":"Lect. Notes Comput. Sci."},{"key":"ref_12","unstructured":"Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., and Wang, W. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1137","DOI":"10.1109\/TPAMI.2016.2577031","article-title":"Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks","volume":"39","author":"Ren","year":"2011","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Liu, W., Jia, Y., and Sermanet, P. (2015, January 7\u201312). Going deeper with convolutions. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Kumar, M., and Kumar Aggarwal, R. (2016, January 1\u20132). Object detection in adverse situations. Proceedings of the 3rd International Conference on Electrical, Electronics, Engineering Trends, Communication, Optimization and Sciences (EEECOS 2016), Tadepalligudem, Andhrapradesh, India.","DOI":"10.1049\/cp.2016.1551"},{"key":"ref_17","unstructured":"Novak, L. (2017). Vehicle Detection and Pose Estimation for Autonomous Driving. [Master\u2019s Thesis, Czech Technical University]."},{"key":"ref_18","unstructured":"Neumann, L., Karg, M., Zhang, S., Scharfenberger, C., and Piegert, E. (2018, January 2\u20136). NightOwls: A Pedestrians at Night Dataset. Proceedings of the 14th Asian Conference on Computer Vision, Perth, Australia."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Singh, B., and Davis, L.S. (2018, January 18\u201323). An Analysis of Scale Invariance in Object Detection\u2014SNIP. In Proceeding of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00377"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Michaelis, C., Mitzkus, B., Geirhos, R., and Rusak, E. (2019). Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming. arXiv.","DOI":"10.12792\/icisip2019.002"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Volk, G., Mueller, S., von Bernuth, A., Hospach, D., and Bringmann, O. (2019, January 27\u201330). Towards Robust CNN-based Object Detection through Augmentation with Synthetic Rain Variations. Proceedings of the 2019 IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand.","DOI":"10.1109\/ITSC.2019.8917269"},{"key":"ref_22","unstructured":"Ren, L., Yin, H., Ge, W., and Meng, Q. (, January 19\u201321). Environment Influences on Uncertainty of Object Detection for Automated Driving Systems. Proceedings of the 2th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), Huaqiao, China."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Reway, F., Huber, W., and Ribeiro, E.P. (2018, January 12\u201314). Test Methodology for Vision-Based ADAS Algorithms with an Automotive Camera-in-the-Loop. Proceedings of the 2018 IEEE International Conference on Vehicular Electronics and Safety (ICVES), Madrid, Spain.","DOI":"10.1109\/ICVES.2018.8519598"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Weber, Y., and Kanarachos, S. (2019). The Correlation between Vehicle Vertical Dynamics and Deep Learning-Based Visual Target State Estimation: A Sensitivity Study. Sensors, 19.","DOI":"10.3390\/s19224870"},{"key":"ref_25","unstructured":"Guo, J., Kurup, U., and Shah, M. (2019). Is It Safe to Drive? An Overview of Factors, Metrics, and Datasets for Driveability Assessment in Autonomous Driving. IEEE Trans. Intell. Transp. Syst., 1\u201317."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1109\/TIV.2018.2886678","article-title":"Test Your Self-Driving Algorithm: An Overview of Publicly Available Driving Datasets and Virtual Testing Environments","volume":"4","author":"Kang","year":"2019","journal-title":"IEEE Trans. Intell. Veh."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1231","DOI":"10.1177\/0278364913491297","article-title":"Vision Meets Robotics: The KITTI Dataset","volume":"32","author":"Geiger","year":"2013","journal-title":"Int. J. Robot. Res."},{"key":"ref_28","unstructured":"Yu, F., Chen, H., Wang, X., Xian, W., and Chen, Y. (2018). BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning. arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Patil, A., Malla, S., Gang, H., and Checn, Y.T. (2019, January 20\u201324). The H3D Dataset for Full-Surround 3D Multi-Object Detection and Tracking in Crowded Urban Scenes. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.","DOI":"10.1109\/ICRA.2019.8793925"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Caesar, H., Bankiti, V., Lang, A.H., Vora, S., and Liong, V.E. (2020). nuScenes: A multimodal dataset for autonomous driving. arXiv.","DOI":"10.1109\/CVPR42600.2020.01164"},{"key":"ref_31","unstructured":"Kesten, R., Usman, M., Houston, J., Pandya, T., and Nadhamuni, K. (2020, April 10). Lyft Level 5 AV Dataset 2019: Dataset. Available online: https:\/\/level5.lyft.com\/dataset\/."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., and Patnaik, V. (2020). Scalability in Perception for Autonomous Driving: Waymo Open Dataset. arXiv.","DOI":"10.1109\/CVPR42600.2020.00252"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Chang, M.F., Lambert, J., Sangkloy, P., Singh, J., and Bak, S. (2019, January 15\u201320). Argoverse: 3D Tracking and Forecasting with Rich Maps. Proceedings of the 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00895"},{"key":"ref_34","unstructured":"Che, Z., Li, G., Li, T., Jiang, B., and Shi, X. (2019). D2-City: A Large-Scale Dashcam Video Dataset of Diverse Trafic Scenarios. arXiv."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Varma, G., Subramanian, A., Namboodiri, A., Manmohan, C., and Jawahr, C.V. (2018, January 12\u201315). IDD: A Dataset for Exploring Problems of Autonomous Navigation in Unconstrained Environments. Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, USA.","DOI":"10.1109\/WACV.2019.00190"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Pham, Q.H., Sevestre, P., Pahwa, R.S., Zhan, H., and Pang, C.H. (2019). A*3D Dataset: Towards Autonomous Driving in Challenging Enironments. arXiv.","DOI":"10.1109\/ICRA40945.2020.9197385"},{"key":"ref_37","unstructured":"Samek, W., Wiegand, T., and M\u00fcller, K.R. (2017). Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models. arXiv."},{"key":"ref_38","unstructured":"Alvarez-Melis, D., and Jaakkola, T.S. (2018, January 3\u20138). Towards Robust Interpretability with Self-Explaining Neural Networks. Proceedings of the 2018 Annual Conference on Neural Information Processing Systems (NeurIPS), Montr\u00e9al, QC, Canada."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Gilpin, L.H., Bau, D., Yuan, B.Z., Bajwa, A., Specter, M., and Kagal, L. (2018, January 1\u20133). Explaining Explanations: An Overview of Interpretability of Machine Learning. Proceedings of the 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), Turin, Italy.","DOI":"10.1109\/DSAA.2018.00018"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 23\u201327). \u201cWhy Should I Trust You?\u201d Explaining the Predictions of Any Classifier. Proceedings of the KDD 16: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.","DOI":"10.1145\/2939672.2939778"},{"key":"ref_41","unstructured":"Molnar, C. (2020, May 04). Interpretable Machine Learning. Available online: https:\/\/christophm.github.io\/interpretable-ml-book\/."},{"key":"ref_42","unstructured":"Feng, J., Yu, Y., and Zhou, Z.H. (2018, January 3\u20138). Multi-Layered Gradient Boosting Decision Trees. Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, QC, Canada."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"3860","DOI":"10.1109\/LRA.2018.2857402","article-title":"Failing to Learn: Autonomously Identifying Perception Failures for Self-Driving Cars","volume":"3","author":"Ramanagopal","year":"2018","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1111\/j.1600-0587.2012.07348.x","article-title":"Collinearity: A review of methods to deal with it and a simulation study evaluating their performance","volume":"36","author":"Dormann","year":"2012","journal-title":"Ecography"},{"key":"ref_45","unstructured":"Geyer, J., Kassahun, Y., Mahmudi, M., Ricou, X., Durgesh, R., Chung, A.S., Hauswald, L., Pham, V.H., M\u00fchlegg, M., and Dorn, S. (2020). A2D2: Audi Autonomous Driving Dataset. arXiv."},{"key":"ref_46","unstructured":"Bochkovskiy, A., Wang, C.Y., and Liao, H.Y.M. (2020). YOLOv4: Optimal Speed and Accuracy of Object Detection. arXiv."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/13\/3699\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:46:08Z","timestamp":1760175968000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/13\/3699"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,7,1]]},"references-count":46,"journal-issue":{"issue":"13","published-online":{"date-parts":[[2020,7]]}},"alternative-id":["s20133699"],"URL":"https:\/\/doi.org\/10.3390\/s20133699","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,7,1]]}}}