{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,2]],"date-time":"2026-07-02T16:02:15Z","timestamp":1783008135240,"version":"3.54.5"},"reference-count":24,"publisher":"MDPI AG","issue":"22","license":[{"start":{"date-parts":[[2022,11,16]],"date-time":"2022-11-16T00:00:00Z","timestamp":1668556800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Millimeter wave (mmWave) radar poses prosperous opportunities surrounding multiple-object tracking and sensing as a unified system. One of the most challenging aspects of exploiting sensing opportunities with mmWave radar is the labeling of mmWave data so that, in turn, a respective model can be designed to achieve the desired tracking and sensing goals. The labeling of mmWave datasets usually involves a domain expert manually associating radar frames with key events of interest. This is a laborious means of labeling mmWave data. This paper presents a framework for training a mmWave radar with a camera as a means of labeling the data and supervising the radar model. The methodology presented in this paper is compared and assessed against existing frameworks that aim to achieve a similar goal. The practicality of the proposed framework is demonstrated through experimentation in varying environmental conditions. The proposed framework is applied to design a mmWave multi-object tracking system that is additionally capable of classifying individual human motion patterns, such as running, walking, and falling. The experimental findings demonstrate a reliably trained radar model that uses a camera for labeling and supervision that can consistently produce high classification accuracy across environments beyond those in which the model was trained against. The research presented in this paper provides a foundation for future research in unified tracking and sensing systems by alleviating the labeling and training challenges associated with designing a mmWave classification model.<\/jats:p>","DOI":"10.3390\/s22228859","type":"journal-article","created":{"date-parts":[[2022,11,17]],"date-time":"2022-11-17T06:24:42Z","timestamp":1668666282000},"page":"8859","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["A Combined mmWave Tracking and Classification Framework Using a Camera for Labeling and Supervised Learning"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0283-4968","authenticated-orcid":false,"given":"Andre","family":"Pearce","sequence":"first","affiliation":[{"name":"Global Big Data Technologies Center, School of Electrical and Data Engineering, University of Technology Sydney, Sydney 2007, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6102-3762","authenticated-orcid":false,"given":"J. Andrew","family":"Zhang","sequence":"additional","affiliation":[{"name":"Global Big Data Technologies Center, School of Electrical and Data Engineering, University of Technology Sydney, Sydney 2007, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2080-4762","authenticated-orcid":false,"given":"Richard","family":"Xu","sequence":"additional","affiliation":[{"name":"Global Big Data Technologies Center, School of Electrical and Data Engineering, University of Technology Sydney, Sydney 2007, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,11,16]]},"reference":[{"key":"ref_1","first-page":"674","article-title":"Vehicle detection based on information fusion of radar and machine vision","volume":"37","author":"Wang","year":"2015","journal-title":"Automot. Eng."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Zhou, Y., Dong, Y., Hou, F., and Wu, J. (2022). Review on Millimeter-Wave Radar and Camera Fusion Technology. Sustainability, 14.","DOI":"10.3390\/su14095114"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1109\/TITS.2006.888597","article-title":"Vehicle and Guard Rail Detection Using Radar and Vision Data Fusion","volume":"8","author":"Alessandretti","year":"2007","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_4","first-page":"21","article-title":"Multi-sensor information fusion technology and its applications","volume":"42","author":"Zhao","year":"2021","journal-title":"Infrared"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1016\/j.procs.2020.02.015","article-title":"Research on Space Fusion Method of Millimeter Wave Radar and Vision Sensor","volume":"166","author":"Cao","year":"2020","journal-title":"Procedia Comput. Sci."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Song, C., Son, G., Kim, H., Gu, D., Lee, J.H., and Kim, Y. (2017, January 9\u201313). A Novel Method of Spatial Calibration for Camera and 2D Radar Based on Registration. Proceedings of the 2017 6th IIAI International Congress on Advanced Applied Informatics (IIAI-AAI), Hamamatsu, Japan.","DOI":"10.1109\/IIAI-AAI.2017.62"},{"key":"ref_7","first-page":"10","article-title":"Unified Calibration Method for Millimeter-Wave Radar and Machine Vision","volume":"V7","author":"Junlong","year":"2018","journal-title":"Int. J. Eng. Res. Technol."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Liu, X., and Cai, Z. (2010, January 27\u201330). Advanced obstacles detection and tracking by fusing millimeter wave radar and image sensor data. Proceedings of the ICCAS 2010, Gyeonggi-do, Korea.","DOI":"10.1109\/ICCAS.2010.5669740"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Wang, T., Xin, J., and Zheng, N. (2011, January 2\u20134). A Method Integrating Human Visual Attention and Consciousness of Radar and Vision Fusion for Autonomous Vehicle Navigation. Proceedings of the 2011 IEEE Fourth International Conference on Space Mission Challenges for Information Technology, Palo Alto, CA, USA.","DOI":"10.1109\/SMC-IT.2011.15"},{"key":"ref_10","unstructured":"Ji, Z., and Prokhorov, D. (July, January 30). Radar-vision fusion for object classification. Proceedings of the 2008 11th International Conference on Information Fusion, Cologne, Germany."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"391","DOI":"10.1108\/IR-06-2018-0113","article-title":"Radar and vision fusion for the real-time obstacle detection and identification","volume":"46","author":"Zhang","year":"2019","journal-title":"Ind. Robot. Int. J. Robot. Res. Appl."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"941","DOI":"10.1109\/JSTSP.2021.3061270","article-title":"RaDICaL: A Synchronized FMCW Radar, Depth, IMU and RGB Camera Data Dataset With Low-Level FMCW Radar Signals","volume":"15","author":"Lim","year":"2021","journal-title":"IEEE J. Sel. Top. Signal Process."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Wei, Z., Zhang, F., Chang, S., Liu, Y., Wu, H., and Feng, Z. (2022). MmWave Radar and Vision Fusion for Object Detection in Autonomous Driving: A Review. Sensors, 22.","DOI":"10.3390\/s22072542"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"182","DOI":"10.1109\/TITS.2002.802932","article-title":"An obstacle detection method by fusion of radar and motion stereo","volume":"3","author":"Kato","year":"2002","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Guo, X.p., Du, J.s., Gao, J., and Wang, W. (2018, January 18\u201320). Pedestrian Detection Based on Fusion of Millimeter Wave Radar and Vision. Proceedings of the 2018 International Conference on Artificial Intelligence and Pattern Recognition, Beijing, China.","DOI":"10.1145\/3268866.3268868"},{"key":"ref_16","unstructured":"Streubel, R., and Yang, B. (2016, January 5\u20138). Fusion of stereo camera and MIMO-FMCW radar for pedestrian tracking in indoor environments. Proceedings of the 2016 19th International Conference on Information Fusion (FUSION), Heidelberg, Germany."},{"key":"ref_17","first-page":"609","article-title":"Moving Object Tracking Based on Millimeter-wave Radar and Vision Sensor","volume":"21","author":"Huang","year":"2018","journal-title":"J. Appl. Sci. Eng."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Richter, E., Schubert, R., and Wanielik, G. (2008, January 4\u20136). Radar and vision based data fusion - Advanced filtering techniques for a multi object vehicle tracking system. Proceedings of the 2008 IEEE Intelligent Vehicles Symposium, Eindhoven, The Netherlands.","DOI":"10.1109\/IVS.2008.4621245"},{"key":"ref_19","unstructured":"Lim, T.Y., Ansari, A., Major, B., Fontijne, D., Hamilton, M., Gowaikar, R., and Subramanian, S. (2019, January 8\u201314). Radar and camera early fusion for vehicle detection in advanced driver assistance systems. Proceedings of the Machine Learning for Autonomous Driving Workshop at the 33rd Conference on Neural Information Processing Systems, Vancouver, BC, Canada."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Nobis, F., Geisslinger, M., Weber, M., Betz, J., and Lienkamp, M. (2019, January 15\u201317). A Deep Learning-based Radar and Camera Sensor Fusion Architecture for Object Detection. Proceedings of the 2019 Sensor Data Fusion: Trends, Solutions, Applications (SDF), Bonn, Germany.","DOI":"10.1109\/SDF.2019.8916629"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Winterling, T., Lombacher, J., Hahn, M., Dickmann, J., and W\u00f6hler, C. (2017, January 28\u201330). Optimizing labelling on radar-based grid maps using active learning. Proceedings of the 2017 18th International Radar Symposium (IRS), Prague, Czech.","DOI":"10.23919\/IRS.2017.8008123"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1137","DOI":"10.1109\/TPAMI.2016.2577031","article-title":"Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks","volume":"39","author":"Ren","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Girshick, R. (2015, January 7\u201313). Fast R-CNN. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.169"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Fang, H.S., Xie, S., Tai, Y.W., and Lu, C. (2017, January 22\u201329). RMPE: Regional Multi-person Pose Estimation. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.256"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/22\/8859\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:19:18Z","timestamp":1760145558000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/22\/8859"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,16]]},"references-count":24,"journal-issue":{"issue":"22","published-online":{"date-parts":[[2022,11]]}},"alternative-id":["s22228859"],"URL":"https:\/\/doi.org\/10.3390\/s22228859","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,16]]}}}