{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T15:36:00Z","timestamp":1783611360117,"version":"3.55.0"},"reference-count":59,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2023,5,26]],"date-time":"2023-05-26T00:00:00Z","timestamp":1685059200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The reliability of autonomous driving sensing systems impacts the overall safety of the driving system. However, perception system fault diagnosis is currently a weak area of research, with limited attention and solutions. In this paper, we present an information-fusion-based fault-diagnosis method for autonomous driving perception systems. To begin, we built an autonomous driving simulation scenario using PreScan software, which collects information from a single millimeter wave (MMW) radar and a single camera sensor. The photos are then identified and labeled via the convolutional neural network (CNN). Then, we fused the sensory inputs from a single MMW radar sensor and a single camera sensor in space and time and mapped the MMW radar points onto the camera image to obtain the region of interest (ROI). Lastly, we developed a method to use information from a single MMW radar to aid in diagnosing defects in a single camera sensor. As the simulation results show, for missing row\/column pixel failure, the deviation typically falls between 34.11% and 99.84%, with a response time of 0.02 s to 1.6 s; for pixel shift faults, the deviation range is between 0.32% and 9.92%, with a response time of 0 s to 0.16 s; for target color loss, faults have a deviation range of 0.26% to 2.88% and a response time of 0 s to 0.05 s. These results prove the technology is effective in detecting sensor faults and issuing real-time fault alerts, providing a basis for designing and developing simpler and more user-friendly autonomous driving systems. Furthermore, this method illustrates the principles and methods of information fusion between camera and MMW radar sensors, establishing the foundation for creating more complicated autonomous driving systems.<\/jats:p>","DOI":"10.3390\/s23115110","type":"journal-article","created":{"date-parts":[[2023,5,27]],"date-time":"2023-05-27T16:18:43Z","timestamp":1685204323000},"page":"5110","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":23,"title":["Fault Diagnosis of the Autonomous Driving Perception System Based on Information Fusion"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9876-4882","authenticated-orcid":false,"given":"Wenkui","family":"Hou","sequence":"first","affiliation":[{"name":"School of Reliability and Systems Engineering, Beihang University, Beijing 100191, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wanyu","family":"Li","sequence":"additional","affiliation":[{"name":"School of Reliability and Systems Engineering, Beihang University, Beijing 100191, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pengyu","family":"Li","sequence":"additional","affiliation":[{"name":"General Design Department, Beijing Mechanical and Electrical Engineering, Beijing 100005, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,5,26]]},"reference":[{"key":"ref_1","first-page":"106","article-title":"The Development of Automotive Intelligence under the 5G Technology","volume":"8","author":"Cai","year":"2020","journal-title":"Automob. Parts"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Antonante, P., Spivak, D.I., and Carlone, L. (2021, January 27). Monitoring and Diagnosability of Perception Systems. Proceedings of the 2021 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Prague, Czech Republic.","DOI":"10.1109\/IROS51168.2021.9636497"},{"key":"ref_3","first-page":"45","article-title":"Research Progress on Multi-Sensor Information Fusion in Unmanned Driving","volume":"1","author":"Zhou","year":"2022","journal-title":"Automot. Dig."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"6640","DOI":"10.1109\/TITS.2021.3059674","article-title":"Robust Target Recognition and Tracking of Self-Driving Cars with Radar and Camera Information Fusion Under Severe Weather Conditions","volume":"23","author":"Liu","year":"2022","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Goelles, T., Schlager, B., and Muckenhuber, S. (2020). Fault Detection, Isolation, Identification and Recovery (FDIIR) Methods for Automotive Perception Sensors Including a Detailed Literature Survey for Lidar. Sensors, 20.","DOI":"10.3390\/s20133662"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Jiang, Q., Zhang, L., and Meng, D. (2019, January 27). Target Detection Algorithm Based on MMW Radar and Camera Fusion. Proceedings of the 2019 IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand.","DOI":"10.1109\/ITSC.2019.8917504"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Chadwick, S., Maddern, W., and Newman, P. (2019, January 20\u201324). Distant Vehicle Detection Using Radar and Vision. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.","DOI":"10.1109\/ICRA.2019.8794312"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Wang, J.-G., Chen, S.J., Zhou, L.-B., Wan, K.-W., and Yau, W.-Y. (2018, January 18). Vehicle Detection and Width Estimation in Rain by Fusing Radar and Vision. Proceedings of the 2018 15th International Conference on Control, Automation, Robotics and Vision (ICARCV), Singapore.","DOI":"10.1109\/ICARCV.2018.8581246"},{"key":"ref_9","first-page":"34","article-title":"Object Detection and Positioning Method Based on Infrared Vision\/Lidar Fusion","volume":"3","author":"Zheng","year":"2021","journal-title":"Navig. Position. Timing"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"2075","DOI":"10.1109\/TITS.2016.2533542","article-title":"On-Road Vehicle Detection and Tracking Using MMW Radar and Monovision Fusion","volume":"17","author":"Wang","year":"2016","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_11","unstructured":"Yang, J. (2019). Pose Tracking and Path Planning for UAV Based on Multi-Sensor Fusion. [Master\u2019s Thesis, Zhejiang University]."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Nabati, R., and Qi, H. (2021, January 3\u20138). Center Fusion: Center-Based Radar and Camera Fusion for 3D Object Detection. Proceedings of the 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA.","DOI":"10.1109\/WACV48630.2021.00157"},{"key":"ref_13","first-page":"53","article-title":"Vehicle radar data fusion target tracking algorithm with multi-mode switching","volume":"11","author":"Zhou","year":"2021","journal-title":"Laser J."},{"key":"ref_14","first-page":"1","article-title":"Design and implementation of vehicle trajectory perception with multi-sensor information fusion","volume":"1","author":"Zhao","year":"2022","journal-title":"Electron. Des. Eng."},{"key":"ref_15","first-page":"498","article-title":"Front vehicle detection based on multi-sensor information fusion","volume":"6","author":"Jia","year":"2022","journal-title":"Infrared Laser Eng."},{"key":"ref_16","first-page":"101","article-title":"Obstacle Avoidance Algorithm for Unmanned Aerial Vehicle Vision Based on Deep Learning","volume":"50","author":"Zhang","year":"2022","journal-title":"J. South China Univ. Technol. Nat. Sci. Ed."},{"key":"ref_17","first-page":"1262","article-title":"An Autonomous Navigation Systems of UAVs Based on Binocular Vision","volume":"52","author":"Hou","year":"2019","journal-title":"J. Tianjin Univ. Sci. Technol."},{"key":"ref_18","first-page":"280","article-title":"Research and Application of Obstacle Avoidance Method Based on Multi-sensor for UAV","volume":"1","author":"Yang","year":"2019","journal-title":"Comput. Meas. Control"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"638","DOI":"10.1016\/j.ymssp.2016.08.028","article-title":"Nonlinear Sensor Fault Diagnosis Using Mixture of Probabilistic PCA Models","volume":"85","author":"Sharifi","year":"2017","journal-title":"Mech. Syst. Signal Process."},{"key":"ref_20","first-page":"528","article-title":"Fault Diagnosis for Sensors in Automotive Li-Ion Power Battery Based on SVD-UKF","volume":"4","author":"Meng","year":"2022","journal-title":"Chin. J. Automot. Eng."},{"key":"ref_21","first-page":"16","article-title":"Aero-engine Sensor Fault Diagnosis and Real-time Verification Based on ARMA Model","volume":"1","author":"Zhao","year":"2022","journal-title":"Aeronaut. Comput. Technol."},{"key":"ref_22","first-page":"1007","article-title":"Accelerometer Fault Diagnosis with Weighted PCA Residual Space","volume":"5","author":"Li","year":"2021","journal-title":"J. Vib. Meas. Diagn."},{"key":"ref_23","first-page":"2257","article-title":"Fault diagnosis method of hydraulic condition monitoring system based on information entropy","volume":"8","author":"Wang","year":"2021","journal-title":"Comput. Eng. Des."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"100935","DOI":"10.1016\/j.jobe.2019.100935","article-title":"Sensor Data Validation and Fault Diagnosis Using Auto-Associative Neural Network for HVAC Systems","volume":"27","author":"Elnour","year":"2020","journal-title":"J. Build. Eng."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1016\/j.neucom.2018.08.046","article-title":"A Hybrid Feature Model and Deep Learning Based Fault Diagnosis for Unmanned Aerial Vehicle Sensors","volume":"319","author":"Guo","year":"2018","journal-title":"Neurocomputing"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"110407","DOI":"10.1016\/j.measurement.2021.110407","article-title":"MLPC-CNN: A Multi-Sensor Vibration Signal Fault Diagnosis Method under Less Computing Resources","volume":"188","author":"Zhang","year":"2022","journal-title":"Measurement"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"96967","DOI":"10.1109\/ACCESS.2022.3205105","article-title":"Fault Diagnosis of Aeroengine Control System Sensor Based on Optimized and Fused Multidomain Feature","volume":"10","author":"Li","year":"2022","journal-title":"IEEE Access"},{"key":"ref_28","unstructured":"Guo, X., Luo, Y., Wang, L., Liu, J., Liao, F., and You, D. (2022). Fault self-diagnosis of structural vibration monitoring sensor and monitoring data recovery based on CNN and DCGAN. J. Railw. Sci. Eng., accepted."},{"key":"ref_29","unstructured":"Zhang, S., and Zhang, T. (2022, January 25). Sensor fault diagnosis method based on CGA-LSTM. Proceedings of the 13th China Satellite Navigation Annual Conference, Beijing, China."},{"key":"ref_30","first-page":"1245","article-title":"Multi-Source Sensor Fault Diagnosis Method Based on Improved CNN-GRU Network","volume":"12","author":"Ma","year":"2021","journal-title":"Trans. Beijing Inst. Technol."},{"key":"ref_31","first-page":"845","article-title":"Sensor fault diagnosis and data reconstruction based on improved LSTM-RF algorithm","volume":"5","author":"Lin","year":"2021","journal-title":"Comput. Eng. Sci."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Realpe, M., Vintimilla, B.X., and Vlacic, L. (2016, January 27\u201329). A Fault Tolerant Perception System for Autonomous Vehicles. Proceedings of the 2016 35th Chinese Control Conference (CCC), Chengdu, China.","DOI":"10.1109\/ChiCC.2016.7554385"},{"key":"ref_33","first-page":"88","article-title":"Object Re-Identification Algorithm Based on Weighted Euclidean Distance Metric","volume":"9","author":"Tan","year":"2015","journal-title":"J. South China Univ. Technol. Nat. Sci. Ed."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"223","DOI":"10.1007\/s42154-021-00172-y","article-title":"A Review of Testing Object-Based Environment Perception for Safe Automated Driving","volume":"5","author":"Hoss","year":"2022","journal-title":"Automot. Innov."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Delecki, H., Itkina, M., Lange, B., Senanayake, R., and Kochenderfer, M.J. (2022, January 23\u201327). How Do We Fail? Stress Testing Perception in Autonomous Vehicles. Proceedings of the 2022 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Kyoto, Japan.","DOI":"10.1109\/IROS47612.2022.9981724"},{"key":"ref_36","first-page":"11","article-title":"Mathematical Techniques in Multisensor Data Fusion","volume":"13","author":"Wang","year":"1995","journal-title":"Biomed. Eng."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1016\/j.inffus.2011.08.001","article-title":"Multisensor Data Fusion: A Review of the State-of-the-Art","volume":"14","author":"Khaleghi","year":"2013","journal-title":"Inf. Fusion"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Wei, Z., Zhang, F., Chang, S., Liu, Y., Wu, H., and Feng, Z. (2022). MmWave Radar and Vision Fusion for Object Detection in Autonomous Driving: A Review. Sensors, 22.","DOI":"10.3390\/s22072542"},{"key":"ref_39","first-page":"181","article-title":"License Plate Recognition Algorithm Based on Deep Learning Model LeNet-5-L","volume":"6","author":"Tao","year":"2021","journal-title":"Comput. Meas. Control"},{"key":"ref_40","first-page":"23","article-title":"RunPool: A Dynamic Pooling Layer for Convolution Neural Network","volume":"13","author":"Wanda","year":"2020","journal-title":"Int. J. Comput. Intell. Syst."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"11201","DOI":"10.1038\/s41598-022-14805-7","article-title":"Spectral pruning of fully connected layers","volume":"12","author":"Buffoni","year":"2022","journal-title":"Sci. Rep."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"103501","DOI":"10.1016\/j.cviu.2022.103501","article-title":"Encoder and Decoder Network with ResNet-50 and Global Average Feature Pooling for Local Change Detection","volume":"222","author":"Panda","year":"2022","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"188","DOI":"10.1016\/j.neucom.2016.11.023","article-title":"G-MS2F: GoogLeNet Based Multi-Stage Feature Fusion of Deep CNN for Scene Recognition","volume":"225","author":"Tang","year":"2017","journal-title":"Neurocomputing"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"012096","DOI":"10.1088\/1755-1315\/619\/1\/012096","article-title":"Research on Vehicle Trajectory Tracking Control in Expressway Maintenance Work Area Based on Coordinate Calibration","volume":"619","author":"Yong","year":"2020","journal-title":"IOP Conf. Ser. Earth Environ. Sci."},{"key":"ref_45","unstructured":"Lyu, D. (2020). Research and Implementation of Infrared Binocular Camera Calibration Method. [Master\u2019s Thesis, Dalian University of Technology]."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Hua, J., and Zeng, L. (2021). Hand\u2013Eye Calibration Algorithm Based on an Optimized Neural Network. Actuators, 10.","DOI":"10.3390\/act10040085"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"111206","DOI":"10.1016\/j.oceaneng.2022.111206","article-title":"Trajectory Tracking Control for Autonomous Underwater Vehicle Based on Rotation Matrix Attitude Representation","volume":"252","author":"Zhu","year":"2022","journal-title":"Ocean Eng."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1515\/spma-2019-0010","article-title":"Construction of Generalized Rotations and Quasi-Orthogonal Matrices","volume":"7","year":"2019","journal-title":"Spec. Matrices"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Girshick, R. (2015, January 7\u201313). Fast R-CNN. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.169"},{"key":"ref_50","unstructured":"Ren, S., He, K., Girshick, R., and Sun, J. (2015). Proceedings of the Advances in Neural Information Processing Systems, Curran Associates, Inc."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (July, January 26). You Only Look Once: Unified, Real-Time Object Detection. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.91"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Ayaz, H., Asgher, U., and Paletta, L. (2021). Proceedings of the Advances in Neuroergonomics and Cognitive Engineering, Springer International Publishing.","DOI":"10.1007\/978-3-030-80285-1"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"5749","DOI":"10.1109\/ACCESS.2017.2731804","article-title":"Robust Lane-Mark Extraction for Autonomous Driving Under Complex Real Conditions","volume":"6","author":"Xuan","year":"2018","journal-title":"IEEE Access"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. (2020, January 13\u201319). NuScenes: A Multimodal Dataset for Autonomous Driving. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01164"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., and Caine, B. (2020, January 13\u201319). Scalability in Perception for Autonomous Driving: Waymo Open Dataset. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00252"},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"1341","DOI":"10.1109\/TITS.2020.2972974","article-title":"Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges","volume":"22","author":"Feng","year":"2021","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Fayyad, J., Jaradat, M.A., Gruyer, D., and Najjaran, H. (2020). Deep Learning Sensor Fusion for Autonomous Vehicle Perception and Localization: A Review. Sensors, 20.","DOI":"10.3390\/s20154220"},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"543","DOI":"10.1016\/j.ins.2017.04.048","article-title":"Hybrid Conditional Random Field Based Camera-LIDAR Fusion for Road Detection","volume":"432","author":"Xiao","year":"2018","journal-title":"Inf. Sci."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"8992","DOI":"10.3390\/s110908992","article-title":"Integrating MMW Radar with a Monocular Vision Sensor for On-Road Obstacle Detection Applications","volume":"11","author":"Wang","year":"2011","journal-title":"Sensors"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/11\/5110\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:43:12Z","timestamp":1760125392000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/11\/5110"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,26]]},"references-count":59,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2023,6]]}},"alternative-id":["s23115110"],"URL":"https:\/\/doi.org\/10.3390\/s23115110","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,26]]}}}