{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,19]],"date-time":"2026-03-19T14:45:59Z","timestamp":1773931559060,"version":"3.50.1"},"reference-count":51,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2021,3,11]],"date-time":"2021-03-11T00:00:00Z","timestamp":1615420800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>The development of autonomous vehicles and unmanned aerial vehicles has led to a current research focus on improving the environmental perception of automation equipment. The unmanned platform detects its surroundings and then makes a decision based on environmental information. The major challenge of environmental perception is to detect and classify objects precisely; thus, it is necessary to perform fusion of different heterogeneous data to achieve complementary advantages. In this paper, a robust object detection and classification algorithm based on millimeter-wave (MMW) radar and camera fusion is proposed. The corresponding regions of interest (ROIs) are accurately calculated from the approximate position of the target detected by radar and cameras. A joint classification network is used to extract micro-Doppler features from the time-frequency spectrum and texture features from images in the ROIs. A fusion dataset between radar and camera is established using a fusion data acquisition platform and includes intersections, highways, roads, and playgrounds in schools during the day and at night. The traditional radar signal algorithm, the Faster R-CNN model and our proposed fusion network model, called RCF-Faster R-CNN, are evaluated in this dataset. The experimental results indicate that the mAP(mean Average Precision) of our network is up to 89.42% more accurate than the traditional radar signal algorithm and up to 32.76% higher than Faster R-CNN, especially in the environment of low light and strong electromagnetic clutter.<\/jats:p>","DOI":"10.3390\/rs13061064","type":"journal-article","created":{"date-parts":[[2021,3,11]],"date-time":"2021-03-11T05:38:22Z","timestamp":1615441102000},"page":"1064","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":55,"title":["Research of Target Detection and Classification Techniques Using Millimeter-Wave Radar and Vision Sensors"],"prefix":"10.3390","volume":"13","author":[{"given":"Zhangjing","family":"Wang","sequence":"first","affiliation":[{"name":"School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3922-117X","authenticated-orcid":false,"given":"Xianhan","family":"Miao","sequence":"additional","affiliation":[{"name":"School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhen","family":"Huang","sequence":"additional","affiliation":[{"name":"School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Haoran","family":"Luo","sequence":"additional","affiliation":[{"name":"School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,3,11]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Nobis, F., Geisslinger, M., Weber, M., Betz, J., and Lienkamp, M. (2019, January 15\u201317). A Deep Learning-Based Radar and Camera Sensor Fusion Architecture for Object Detection. Proceedings of the 2019 Sensor Data Fusion: Trends, Solutions, Applications (SDF), Bonn, DE, USA.","DOI":"10.1109\/SDF.2019.8916629"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1109\/MGRS.2019.2937630","article-title":"Linking Points with Labels in 3D: A Review of Point Cloud Semantic Segmentation","volume":"8","author":"Xie","year":"2020","journal-title":"IEEE Geosci. Remote. Sens. Mag."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Guo, X.-P., Du, J.-S., Gao, J., and Wang, W. (2018). Pedestrian Detection Based on Fusion of Millimeter Wave Radar and Vision. Proceedings of the 2018 International Conference on Artificial Intelligence and Pattern Recognition, Association for Computing Machinery.","DOI":"10.1145\/3268866.3268868"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Zewge, N.S., Kim, Y., Kim, J., and Kim, J.-H. (2019, January 1\u20133). Millimeter-Wave Radar and RGB-D Camera Sensor Fusion for Real-Time People Detection and Tracking. Proceedings of the 2019 7th International Conference on Robot Intelligence Technology and Applications (RiTA), Daejeon, Korea.","DOI":"10.1109\/RITAPP.2019.8932892"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Guo, Y., Wang, H., Hu, Q., Liu, H., Liu, L., and Bennamoun, M. (2020). Deep Learning for 3d Point Clouds: A survey. IEEE Trans. Pattern Anal. Mach. Intell., 1.","DOI":"10.1109\/TPAMI.2020.3005434"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"2293","DOI":"10.1177\/0954407019867492","article-title":"A Multi-Sensor Fusion and Object Tracking Algorithm for Self-Driving Vehicles. Proceedings of the Institution of Mechanical Engineers","volume":"233","author":"YI","year":"2019","journal-title":"Part D J. Automob. Eng."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1016\/j.trf.2016.12.002","article-title":"A Real-Time Multisensor Fusion Verification Framework for Advanced Driver Assistance Systems","volume":"61","author":"Elgharbawy","year":"2019","journal-title":"Transp. Res. Part F Traffic Psychol. Behav."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"101289","DOI":"10.1016\/j.cogpsych.2020.101289","article-title":"A Diffusion Model Analysis of Target Detection in Near-Threshold Visual Search","volume":"120","author":"Corbett","year":"2020","journal-title":"Cogn. Psychol."},{"key":"ref_9","unstructured":"Zou, Z., Shi, Z., Guo, Y., and Ye, J. (2019). Object Detection in 20 Years: A Survey. arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"675","DOI":"10.1631\/FITEE.1900518","article-title":"A Survey on Multi-Sensor Fusion Based Obstacle Detection for Intelligent Ground Vehicles in Off-Road Environments","volume":"21","author":"Hu","year":"2020","journal-title":"Front. Inf. Technol. Electron. Eng."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"2847","DOI":"10.1109\/ACCESS.2019.2962554","article-title":"Multi-Sensor Fusion in Automated Driving: A. Aurvey","volume":"8","author":"Wang","year":"2019","journal-title":"IEEE Access"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"012059","DOI":"10.1088\/1742-6596\/1314\/1\/012059","article-title":"Research on Information Fusion Method of Millimeter Wave Radar and Monocular Camera for Intelligent Vehicle","volume":"1314","author":"Feng","year":"2019","journal-title":"J. Phys. Conf. Ser."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Steinbaeck, J., Steger, C., Brenner, E., Holweg, G., and Druml, N. (2019, January 28\u201330). Occupancy Grid Fusion of Low-Level Radar and Time-of-Flight Sensor Data. Proceedings of the: 2019 22nd Euromicro Conference on Digital System Design (DSD), Kallithea, Greece.","DOI":"10.1109\/DSD.2019.00038"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"7283","DOI":"10.1109\/JSEN.2019.2914365","article-title":"Human Target Detection, Tracking, and Classification Using 24-GHz FMCW Radar","volume":"19","author":"Will","year":"2019","journal-title":"IEEE Sens. J."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Chen, B., Pei, X., and Chen, Z. (2020). Research on Target Detection Based on Distributed Track Fusion for Intelligent Vehicles. Sensors, 20.","DOI":"10.3390\/s20010056"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Kim, D., and Kim, S. (2019, January 15\u201318). Extrinsic Parameter Calibration of 2D Radar-Camera Using Point Matching and Generative Optimization. Proceedings of the 2019 19th International Conference on Control, Automation and Systems (ICCAS), Jeju, Korea.","DOI":"10.23919\/ICCAS47443.2019.8971568"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Palffy, A., Kooij, J.F.P., and Gavrila, D.M. (2019, January 9\u201312). Occlusion Aware Sensor Fusion for Early Crossing Pedestrian Detection. Proceedings of the 2019 IEEE Intelligent Vehicles Symposium (IV), Paris, France.","DOI":"10.1109\/IVS.2019.8814065"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Chang, S., Zhang, Y., Zhao, X., Huang, S., Feng, Z., Wei, Z., and Zhang, F. (2020). Spatial Attention Fusion for Obstacle Detection Using MmWave Radar and Vision Sensor. Sensors, 20.","DOI":"10.3390\/s20040956"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Yang, B., Guo, R., Liang, M., Casas, S., and Urtasun, R. (2020). Radarnet: Exploiting Radar for Robust Perception of Dynamic Objects. Proceedings of the European Conference on Computer Vision, Springer.","DOI":"10.1007\/978-3-030-58523-5_29"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"012063","DOI":"10.1088\/1742-6596\/1314\/1\/012063","article-title":"Preceding Vehicle Detection Method Based on Information Fusion of Millimeter Wave Radar and Deep Learning Vision","volume":"1314","author":"Li","year":"2019","journal-title":"J. Phys. Conf. Ser."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1550147719849381","DOI":"10.1177\/1550147719849381","article-title":"The Generalization Negation of Probability Distribution and its Application in Target Recognition Based on Sensor Fusion","volume":"15","author":"Gao","year":"2019","journal-title":"Int. J. Distrib. Sens. Netw."},{"key":"ref_22","first-page":"1608","article-title":"Camera-Radar Data Fusion for Target Detection via Kalman Filter and Bayesian Estimation","volume":"1","author":"Yu","year":"2018","journal-title":"SAE Tech. Pap."},{"key":"ref_23","first-page":"613","article-title":"Study on Target Tracking Based on Vision and Radar Sensor Fusion","volume":"1","author":"Wu","year":"2018","journal-title":"SAE Tech. Pap."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1137","DOI":"10.1109\/TPAMI.2016.2577031","article-title":"Faster R-Cnn: Towards Real-Time Object Detection with Region Proposal Networks","volume":"39","author":"Ren","year":"2016","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"391","DOI":"10.1108\/IR-06-2018-0113","article-title":"Radar and Vision Fusion for the Real-Time Obstacle Detection and Identification","volume":"46","author":"Zhang","year":"2019","journal-title":"Ind. Robot. Int. J. Robot. Res. Appl."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Koci\u0107, J., Nenad, J., and Vujo, D. (2018, January 20\u201321). Sensors and Sensor Fusion in Autonomous Vehicles. Proceedings of the 2018 26th Telecommunications Forum (TELFOR), Belgrade, Serbia.","DOI":"10.1109\/TELFOR.2018.8612054"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Zhou, X., Qian, L.-C., You, P.-J., Ding, Z.-G., and Han, Y.-Q. (2018, January 23\u201327). Fall Detection Using Convolutional Neural Network with Multi-Sensor Fusion. Proceedings of the 2018 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), San Diego, CA, USA.","DOI":"10.1109\/ICMEW.2018.8551564"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Sengupta, A., Feng, J., and Siyang, C. (2019, January 15\u201319). A Dnn-LSTM based target tracking approach using mmWave radar and camera sensor fusion. Proceedings of the 2019 IEEE National Aerospace and Electronics Conference (NAECON), Dayton, OH, USA.","DOI":"10.1109\/NAECON46414.2019.9058168"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Jha, H., Vaibhav, L., and Debashish, C. (2019, January 7\u20138). Object Detection and Identification Using Vision and Radar Data Fusion System for Ground-Based Navigation. Proceedings of the2019 6th International Conference on Signal Processing and Integrated Networks (SPIN), Noida, India.","DOI":"10.1109\/SPIN.2019.8711717"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Ulrich, M., Hess, T., Abdulatif, S., and Yang, B. (2018, January 10\u201313). Person Recognition Based on Micro-Doppler and Thermal Infrared Camera Fusion for Firefighting. Proceedings of the 2018 21st International Conference on Information Fusion (FUSION), Cambridge, UK.","DOI":"10.23919\/ICIF.2018.8455723"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"258-1","DOI":"10.2352\/ISSN.2470-1173.2018.17.AVM-258","article-title":"Camera Radar Fusion for Increased Reliability in ADAS Applications","volume":"2018","author":"Zhong","year":"2018","journal-title":"Electron. Imaging"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Jibrin, F.A., Zhenmiao, D., and Yixiong, Z. (2019, January 10\u201313). An Object Detection and Classification Method using Radar and Camera Data Fusion. Proceedings of the 2019 IEEE International Conference on Signal, Information and Data Processing (ICSIDP), Chongqing, China.","DOI":"10.1109\/ICSIDP47821.2019.9173452"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1357","DOI":"10.1109\/TAES.2019.2929974","article-title":"Joint Registration and Fusion of an Infrared Camera and Scanning Radar in a Maritime Context","volume":"56","author":"Cormack","year":"2019","journal-title":"IEEE Trans. Aerosp. Electron. Syst."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"75223","DOI":"10.1109\/ACCESS.2020.2985075","article-title":"Camera and Radar Sensor Fusion for Robust Vehicle Localization via Vehicle Part Localization","volume":"8","author":"Kang","year":"2020","journal-title":"IEEE Access"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Dimitrievski, M., Jacobs, L., Veelaert, P., and Philips, W. (2019, January 27\u201330). People Tracking by Cooperative Fusion of RADAR and Camera Sensors. Proceedings of the 2019 IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand.","DOI":"10.1109\/ITSC.2019.8917238"},{"key":"ref_36","unstructured":"Nabati, R., and Hairong, Q. (2020). Radar-Camera Sensor Fusion for Joint Object Detection and Distance Estimation in Autonomous Vehicles. arXiv."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Jiang, Q., Lijun, Z., and Dejian, M. (2019, January 27\u201330). Target Detection Algorithm Based on MMW Radar and Camera Fusion. Proceedings of the 2019 IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand.","DOI":"10.1109\/ITSC.2019.8917504"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"137065","DOI":"10.1109\/ACCESS.2019.2942382","article-title":"Extending Reliability of mmWave Radar Tracking and Detection via Fusion with Camera","volume":"7","author":"Zhang","year":"2019","journal-title":"IEEE Access"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"8079","DOI":"10.1109\/JSEN.2019.2917375","article-title":"Human Activity Detection and Coarse Localization Outdoors Using Micro-Doppler Signatures","volume":"19","author":"Luo","year":"2019","journal-title":"IEEE Sens. J."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"304","DOI":"10.1016\/j.eswa.2019.06.048","article-title":"Pedestrian Recognition Using Micro Doppler Effects of Radar Signals Based on Machine Learning and Multi-Objective Optimization","volume":"136","author":"Severino","year":"2019","journal-title":"Expert Syst. Appl."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"24122","DOI":"10.1109\/ACCESS.2019.2900303","article-title":"Using Micro-Doppler Radar to Measure Gait Features Associated with Cognitive Functions in Elderly Adults","volume":"7","author":"Saho","year":"2019","journal-title":"IEEE Access"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"3197","DOI":"10.1109\/TAES.2020.2969579","article-title":"Motion Classification Using Kinematically Sifted Acgan-Synthesized Radar Micro-Doppler Signatures","volume":"56","author":"Erol","year":"2020","journal-title":"IEEE Trans. Aerosp. Electron. Syst."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.cviu.2019.04.002","article-title":"Automotive Radar and Camera Fusion Using Generative Adversarial Networks","volume":"184","author":"Lekic","year":"2019","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"396","DOI":"10.1109\/LGRS.2019.2919770","article-title":"Generative Adversarial Networks for Classification of Micro-Doppler Signatures of Human Activity","volume":"17","author":"Alnujaim","year":"2019","journal-title":"IEEE Geosci. Remote. Sens. Lett."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Nabati, R., and Hairong, Q. (2020). CenterFusion: Center-based Radar and Camera Fusion for 3D Object Detection. arXiv.","DOI":"10.1109\/WACV48630.2021.00157"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Yu, H., Zhang, F., Huang, P., Wang, C., and Yuanhao, L. (2020, January 25\u201329). Autonomous Obstacle Avoidance for UAV based on Fusion of Radar and Monocular Camera. Proceedings of the J International Conference on Intelligent Robots and System, Las Vegas, NV, USA.","DOI":"10.1109\/IROS45743.2020.9341432"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Samaras, S., Diamantidou, E., Ataloglou, D., Sakellariou, N., Vafeiadis, A., Magoulianitis, V., Lalas, A., Dimou, A., Zarpalas, D., and Votis, K. (2019). Deep Learning on Multi Sensor Data for Counter UAV Applications\u2014A Systematic Review. Sensors, 19.","DOI":"10.3390\/s19224837"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Jovanoska, S., Martina, B., and Wolfgang, K. (2018, January 20\u201322). Multisensor Data Fusion for UAV Detection and Tracking. Proceedings of the2018 19th International Radar Symposium (IRS), Bonn, Germany.","DOI":"10.23919\/IRS.2018.8447971"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Wang, C., Wang, Z., Yu, Y., and Miao, X. (2019, January 23\u201326). Rapid Recognition of Human Behavior Based on Micro-Doppler Feature. Proceedings of the 2019 International Conference on Control, Automation and Information Sciences (ICCAIS), Chengdu, China.","DOI":"10.1109\/ICCAIS46528.2019.9074550"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Yu, Y., Wang, Z., Miao, X., and Wang, C. (2019, January 23\u201326). Human Parameter Estimation Based on Sparse Reconstruction. Proceedings of the 2019 International Conference on Control, Automation and Information Sciences (ICCAIS), Chengdu, China.","DOI":"10.1109\/ICCAIS46528.2019.9074593"},{"key":"ref_51","unstructured":"Simonyan, K., and Andrew, Z. (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/13\/6\/1064\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:34:01Z","timestamp":1760160841000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/13\/6\/1064"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,11]]},"references-count":51,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2021,3]]}},"alternative-id":["rs13061064"],"URL":"https:\/\/doi.org\/10.3390\/rs13061064","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,11]]}}}