{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,7]],"date-time":"2026-05-07T16:28:28Z","timestamp":1778171308685,"version":"3.51.4"},"reference-count":30,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2023,3,2]],"date-time":"2023-03-02T00:00:00Z","timestamp":1677715200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Science and Technology Council","award":["111-2218-E-A49-028-"],"award-info":[{"award-number":["111-2218-E-A49-028-"]}]},{"name":"National Science and Technology Council","award":["111-2634-F-A49 -009 -"],"award-info":[{"award-number":["111-2634-F-A49 -009 -"]}]},{"name":"National Science and Technology Council","award":["111-2218-E-002 -039 -"],"award-info":[{"award-number":["111-2218-E-002 -039 -"]}]},{"name":"National Science and Technology Council","award":["111-2221-E-A49 -126 -MY3"],"award-info":[{"award-number":["111-2221-E-A49 -126 -MY3"]}]},{"name":"National Science and Technology Council","award":["110-2634-F-A49 -004 -"],"award-info":[{"award-number":["110-2634-F-A49 -004 -"]}]},{"name":"National Science and Technology Council","award":["110-2221-E-A49 -145 -MY3"],"award-info":[{"award-number":["110-2221-E-A49 -145 -MY3"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>This paper proposes a deep learning-based mmWave radar and RGB camera sensor early fusion method for object detection and tracking and its embedded system realization for ADAS applications. The proposed system can be used not only in ADAS systems but also to be applied to smart Road Side Units (RSU) in transportation systems to monitor real-time traffic flow and warn road users of probable dangerous situations. As the signals of mmWave radar are less affected by bad weather and lighting such as cloudy, sunny, snowy, night-light, and rainy days, it can work efficiently in both normal and adverse conditions. Compared to using an RGB camera alone for object detection and tracking, the early fusion of the mmWave radar and RGB camera technology can make up for the poor performance of the RGB camera when it fails due to bad weather and\/or lighting conditions. The proposed method combines the features of radar and RGB cameras and directly outputs the results from an end-to-end trained deep neural network. Additionally, the complexity of the overall system is also reduced such that the proposed method can be implemented on PCs as well as on embedded systems like NVIDIA Jetson Xavier at 17.39 fps.<\/jats:p>","DOI":"10.3390\/s23052746","type":"journal-article","created":{"date-parts":[[2023,3,3]],"date-time":"2023-03-03T02:03:08Z","timestamp":1677808988000},"page":"2746","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["Deep Learning Derived Object Detection and Tracking Technology Based on Sensor Fusion of Millimeter-Wave Radar\/Video and Its Application on Embedded Systems"],"prefix":"10.3390","volume":"23","author":[{"given":"Jia-Jheng","family":"Lin","sequence":"first","affiliation":[{"name":"Institute of Electronics, Nation Yang Ming Chiao Tung University, Hsinchu 30010, Taiwan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0402-2621","authenticated-orcid":false,"given":"Jiun-In","family":"Guo","sequence":"additional","affiliation":[{"name":"Institute of Electronics, Nation Yang Ming Chiao Tung University, Hsinchu 30010, Taiwan"},{"name":"Pervasive Artificial Intelligence Research (PAIR) Labs, National Yang Ming Chiao Tung University, Hsinchu 30010, Taiwan"},{"name":"Wistron-NCTU Embedded Artificial Intelligence Research Center, National Yang Ming Chiao Tung University, Hsinchu 30010, Taiwan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9434-5899","authenticated-orcid":false,"given":"Vinay Malligere","family":"Shivanna","sequence":"additional","affiliation":[{"name":"Institute of Electronics, Nation Yang Ming Chiao Tung University, Hsinchu 30010, Taiwan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5724-6031","authenticated-orcid":false,"given":"Ssu-Yuan","family":"Chang","sequence":"additional","affiliation":[{"name":"Department of Multimedia, Mediatek Inc., Hsinchu 30010, Taiwan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,3,2]]},"reference":[{"key":"ref_1","unstructured":"Redmon, J., and Farhadi, A. (2018). YOLOv3: An incremental improvement. arXiv."},{"key":"ref_2","unstructured":"Bochkovskiy, A., Wang, C.-Y., and Liao, H.-Y.M. (2020). YOLOv4: Optimal speed and accuracy of object detection. arXiv."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Chang, S., Zhang, Y., Zhang, F., Zhao, X., Huang, S., and Feng, Z. (2020). Spatial attention fusion for obstacle detection using mmWave radar and vision sensor. Sensors, 20.","DOI":"10.3390\/s20040956"},{"key":"ref_4","unstructured":"Lu, J.X., Lin, J.C., Vinay, M.S., Chen, P.-Y., and Guo, J.-I. (2020, January 7\u201310). Fusion technology of radar and RGB camera sensors for object detection and tracking and its embedded system implementation. Proceedings of the 2020 Asia-Pacific Signal and Information Processing As-sociation Annual Summit and Conference (APSIPA ASC), Auckland, New Zealand."},{"key":"ref_5","unstructured":"Obrvan, M., \u0106esi\u0107, J., and Petrovi\u0107, I. (2015). Advances in Intelligent Systems and Computing, Elsevier."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"606","DOI":"10.1109\/TITS.2009.2032769","article-title":"Collision sensing by stereo vision and radar sensor fusion","volume":"10","author":"Wu","year":"2009","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"81564","DOI":"10.1109\/ACCESS.2021.3083503","article-title":"A Novel Multi-Sensor Fusion Based Object Detection and Recognition Algorithm for Intelligent Assisted Driving","volume":"9","author":"Liu","year":"2021","journal-title":"IEEE Access"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Jha, H., Lodhi, V., and Chakravarty, D. (2019, January 7\u20138). Object Detection and Identification Using Vision and Radar Data Fusion System for Ground-Based Navigation. Proceedings of the 2019 6th International Conference on Signal Processing and Integrated Net-works (SPIN), Noida, India.","DOI":"10.1109\/SPIN.2019.8711717"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Kim, K.-E., Lee, C.-J., Pae, D.-S., and Lim, M.-T. (2017, January 18\u201321). Sensor fusion for vehicle tracking with camera and radar sensor. Proceedings of the 2017 17th International Conference on Control, Automation and Systems (ICCAS), Jeju, Republic of Korea.","DOI":"10.23919\/ICCAS.2017.8204375"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"8992","DOI":"10.3390\/s110908992","article-title":"Integrating Millimeter Wave Radar with a Monocular Vision Sensor for On-Road Obstacle Detection Applications","volume":"11","author":"Wang","year":"2011","journal-title":"Sensors"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Guo, X., Du, J., Gao, J., and Wang, W. (2018, January 18\u201320). Pedestrian detection based on fusion of millimeter wave radar and vision. Proceedings of the 2018 International Conference on Artificial Intelligence and Pattern Recognition, Beijing, China.","DOI":"10.1145\/3268866.3268868"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"2075","DOI":"10.1109\/TITS.2016.2533542","article-title":"On-road vehicle detection and tracking using MMW radar and monovision fusion","volume":"17","author":"Wang","year":"2016","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Chadwick, S., Maddern, W., and Newman, P. (2019, January 20\u201324). Distant vehicle detection using radar and vision. Proceedings of the International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.","DOI":"10.1109\/ICRA.2019.8794312"},{"key":"ref_14","unstructured":"John, V., and Mita, S. (2019). Pacific-Rim Symposium on Image and Video Technology, Springer."},{"key":"ref_15","unstructured":"Geisslinger, M., Weber, M., Betz, J., and Lienkamp, M. (2019, January 15\u201317). A deep learning-based radar and camera sensor fusion architecture for object detection. Proceedings of the 2019 Sensor Data Fusion: Trends, Solutions, Applications (SDF), Bonn, Germany."},{"key":"ref_16","unstructured":"Yun, S., Han, D., Joon Oh, S., Chun, S., Choe, J., and Yoo, Y. (November, January 27). CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features. Proceedings of the International Conference on Computer Vision (ICCV), Seoul, Republic of Korea."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"012094","DOI":"10.1088\/1742-6596\/1684\/1\/012094","article-title":"Improved Mosaic: Algorithms for more Complex Images","volume":"1684","author":"Hao","year":"2020","journal-title":"J. Phys. Conf. Ser."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C.L., and Doll\u00e1r, P. (2014, January 6\u201312). Microsoft COCO: Common Objects in Context. Proceedings of the European Conference on Computer Vision (ECCV), Cham, Germany.","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Wang, C.-Y., Liao, H.-Y.M., Wu, Y.-H., Chen, P.-Y., Hsieh, J.-W., and Yeh, I.-H. (2020, January 14\u201319). CSPNet: A New Backbone That Can Enhance Learning Capability of CNN. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPRW), Seattle, WA, USA.","DOI":"10.1109\/CVPRW50498.2020.00203"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"3519","DOI":"10.1109\/TGRS.2007.906140","article-title":"Ligthart. Signal processing for FMCW SAR","volume":"45","author":"Meta","year":"2007","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_21","first-page":"651","article-title":"Data clustering: 50 years beyond K-means. Pattern Recognition Letters","volume":"31","author":"Jain","year":"2010","journal-title":"Corrected Proof"},{"key":"ref_22","unstructured":"Ester, M., Kriegel, H.-P., Sander, J., and Xiaowei, X. (1996, January 2\u20134). A density-based algorithm for discovering clusters in large spatial databases with noise. Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining, Portland, OR, USA."},{"key":"ref_23","first-page":"289","article-title":"Unified calibration method for millimeter-wave radar and camera","volume":"54","author":"Luo","year":"2014","journal-title":"J. Tsinghua Univ. Sci. Technol."},{"key":"ref_24","first-page":"298","article-title":"Landmark based shortest path detection by using A* and Haversine formula","volume":"1","author":"Chopde","year":"2013","journal-title":"Int. J. Innov. Res. Comput. Commun. Eng."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Taguchi, G., and Jugulum, R. (2002). The Mahalanobis-Taguchi Strategy: A Pattern Technology System, John Wiley & Sons.","DOI":"10.1002\/9780470172247"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"7380","DOI":"10.1109\/TPAMI.2021.3119563","article-title":"Detection and Tracking Meet Drones Challenge","volume":"44","author":"Zhu","year":"2021","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Ma, K., Zhang, H., Wang, R., and Zhang, Z. (2017, January 15\u201317). Target tracking system for multi-sensor data fusion. Proceedings of the 2017 IEEE 2nd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC), Chengdu, China.","DOI":"10.1109\/ITNEC.2017.8285099"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"6640","DOI":"10.1109\/TITS.2021.3059674","article-title":"Robust Target Recognition and Tracking of Self-Driving Cars With Radar and Camera Information Fusion Under Severe Weather Conditions","volume":"23","author":"Liu","year":"2021","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_29","unstructured":"(2022, July 23). Texas Instruments. IWR6843: Single-Chip 60-GHz to 64-GHz Intelligent mmWave Sensor Integrating Processing Capability. Available online: https:\/\/www.ti.com\/product\/IWR6843."},{"key":"ref_30","unstructured":"NVIDIA (2022, July 24). NVIDIA Jetson AGX Xavier: The AI Platform for Autonomous Machines. Available online: https:\/\/developer.nvidia.com\/embedded\/jetson-agx-xavier-developer-kit."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/5\/2746\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:46:19Z","timestamp":1760121979000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/5\/2746"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,2]]},"references-count":30,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2023,3]]}},"alternative-id":["s23052746"],"URL":"https:\/\/doi.org\/10.3390\/s23052746","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,2]]}}}