{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,23]],"date-time":"2026-07-23T16:12:23Z","timestamp":1784823143359,"version":"3.55.0"},"reference-count":32,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2026,1,21]],"date-time":"2026-01-21T00:00:00Z","timestamp":1768953600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Natural Science Foundation Projects of the Jiangxi Provincial Department of Education","award":["20252BAC240200"],"award-info":[{"award-number":["20252BAC240200"]}]},{"name":"Natural Science Foundation Projects of the Jiangxi Provincial Department of Education","award":["20224BAB202014"],"award-info":[{"award-number":["20224BAB202014"]}]},{"name":"Doctoral Research Start-up Project of Jiangxi University of Water Resources and Electric Power","award":["2022KYQD027"],"award-info":[{"award-number":["2022KYQD027"]}]},{"name":"Jiangxi Provincial Graduate Professional Degree Teaching Case Project","award":["5"],"award-info":[{"award-number":["5"]}]},{"name":"2024 Degree and Graduate Education Teaching Reform Research Project","award":["NGYJG-2024-001"],"award-info":[{"award-number":["NGYJG-2024-001"]}]},{"name":"First-Class Course of Jiangxi University of Water Resources and Electric Power","award":["29"],"award-info":[{"award-number":["29"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Effectively fusing the rich semantic information from camera images with the high-precision geometric measurements provided by LiDAR point clouds is a key challenge in mobile robot environmental perception. To address this problem, this paper proposes a highly extensible instance-aware fusion framework designed to achieve efficient alignment and unified modeling of heterogeneous sensory data. The proposed approach adopts a modular processing pipeline. First, semantic instance masks are extracted from RGB images using an instance segmentation network, and a projection mechanism is employed to establish spatial correspondences between image pixels and LiDAR point cloud measurements. Subsequently, three-dimensional bounding boxes are reconstructed through point cloud clustering and geometric fitting, and a reprojection-based validation mechanism is introduced to ensure consistency across modalities. Building upon this representation, the system integrates a data association module with a Kalman filter-based state estimator to form a closed-loop multi-object tracking framework. Experimental results on the KITTI dataset demonstrate that the proposed system achieves strong 2D and 3D detection performance across different difficulty levels. In multi-object tracking evaluation, the method attains a MOTA score of 47.8 and an IDF1 score of 71.93, validating the stability of the association strategy and the continuity of object trajectories in complex scenes. Furthermore, real-world experiments on a mobile computing platform show an average end-to-end latency of only 173.9 ms, while ablation studies further confirm the effectiveness of individual system components. Overall, the proposed framework exhibits strong performance in terms of geometric reconstruction accuracy and tracking robustness, and its lightweight design and low latency satisfy the stringent requirements of practical robotic deployment.<\/jats:p>","DOI":"10.3390\/s26020718","type":"journal-article","created":{"date-parts":[[2026,1,21]],"date-time":"2026-01-21T13:59:54Z","timestamp":1769003994000},"page":"718","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Robot Object Detection and Tracking Based on Image\u2013Point Cloud Instance Matching"],"prefix":"10.3390","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2231-4153","authenticated-orcid":false,"given":"Hongxing","family":"Wang","sequence":"first","affiliation":[{"name":"Jiangxi Provincial Key Laboratory of Precision Drive and Equipment, Jiangxi University of Water Resources and Electric Power, Nanchang 330099, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rui","family":"Zhu","sequence":"additional","affiliation":[{"name":"Jiangxi Provincial Key Laboratory of Precision Drive and Equipment, Jiangxi University of Water Resources and Electric Power, Nanchang 330099, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zelin","family":"Ye","sequence":"additional","affiliation":[{"name":"School of Mechatronics Engineering, Harbin Institute of Technology, Harbin 150001, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0061-0311","authenticated-orcid":false,"given":"Yaxin","family":"Li","sequence":"additional","affiliation":[{"name":"Fujian Key Laboratory of Intelligent Operation and Maintenance Robot Technology, Fujian (Quanzhou) Institute of Advanced Manufacturing Technology, Quanzhou 362000, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,1,21]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1109\/JPROC.2023.3238524","article-title":"Object detection in 20 years: A survey","volume":"111","author":"Zou","year":"2023","journal-title":"Proc. IEEE"},{"key":"ref_2","unstructured":"Bolya, D., Zhou, C., Xiao, F., and Lee, Y.J. (November, January 27). Yolact: Real-time instance segmentation. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Seoul, Republic of Korea."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Zhao, X., Li, M., Weber, C., Hafez, M.B., and Wermter, S. (2023, January 1\u20135). Chat with the environment: Interactive multimodal perception using large language models. Proceedings of the 2023 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Detroit, MI, USA.","DOI":"10.1109\/IROS55552.2023.10342363"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Chung, S., Lim, J., Noh, K.J., Kim, G., and Jeong, H. (2019). Sensor data acquisition and multimodal sensor fusion for human activity recognition using deep learning. Sensors, 19.","DOI":"10.3390\/s19071716"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Bijelic, M., Gruber, T., Mannan, F., Kraus, F., Ritter, W., Dietmayer, K., and Heide, F. (2020, January 13\u201319). Seeing through fog without seeing fog: Deep multimodal sensor fusion in unseen adverse weather. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01170"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., and Caine, B. (2020, January 13\u201319). Scalability in perception for autonomous driving: Waymo open dataset. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00252"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Bras\u00f3, G., and Leal-Taix\u00e9, L. (2020, January 13\u201319). Learning a neural solver for multiple object tracking. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00628"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Chen, Y.T., Shi, J., Ye, Z., Mertz, C., Ramanan, D., and Kong, S. (2022, January 23\u201327). Multimodal object detection via probabilistic ensembling. Proceedings of the European Conference on Computer Vision (ECCV 2022), Tel Aviv, Israel.","DOI":"10.1007\/978-3-031-20077-9_9"},{"key":"ref_9","unstructured":"Chen, K., Wang, J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., and Xu, J. (2019). MMDetection: Open mmlab detection toolbox and benchmark. arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"He, K., Gkioxari, G., Doll\u00e1r, P., and Girshick, R. (2017, January 22\u201329). Mask r-cnn. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.322"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., and Berg, A.C. (2016, January 11\u201314). Ssd: Single shot multibox detector. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46448-0_2"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Sohan, M., Sai Ram, T., and Rami Reddy, C.V. (2024, January 18\u201320). A review on yolov8 and its advancements. Proceedings of the International Conference on Data Intelligence and Cognitive Informatics, Tirunelveli, India.","DOI":"10.1007\/978-981-99-7962-2_39"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Tian, Z., Shen, C., and Chen, H. (2020, January 23\u201328). Conditional convolutions for instance segmentation. Proceedings of the European Conference on Computer Vision, Glasgow, UK.","DOI":"10.1007\/978-3-030-58452-8_17"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"3069","DOI":"10.1007\/s11263-021-01513-4","article-title":"Fairmot: On the fairness of detection and re-identification in multiple object tracking","volume":"129","author":"Zhang","year":"2021","journal-title":"Int. J. Comput. Vis."},{"key":"ref_15","unstructured":"Aharon, N., Orfaig, R., and Bobrovsky, B.Z. (2022). BoT-SORT: Robust associations multi-pedestrian tracking. arXiv."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"12783","DOI":"10.1109\/TPAMI.2022.3213073","article-title":"Looking beyond two frames: End-to-end multi-object tracking using spatial and temporal transformers","volume":"45","author":"Zhu","year":"2022","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. (2020, January 13\u201319). nuscenes: A multimodal dataset for autonomous driving. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01164"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1341","DOI":"10.1109\/TITS.2020.2972974","article-title":"Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges","volume":"22","author":"Feng","year":"2020","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Rusu, R.B., and Cousins, S. (2011, January 9\u201313). 3d is here: Point cloud library (pcl). Proceedings of the 2011 IEEE International Conference on Robotics and Automation, Shanghai, China.","DOI":"10.1109\/ICRA.2011.5980567"},{"key":"ref_20","first-page":"41","article-title":"Efficient online segmentation for sparse 3D laser scans","volume":"85","author":"Bogoslavskyi","year":"2017","journal-title":"PFG\u2013J. Photogramm. Remote Sens. Geoinf. Sci."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Yan, Y., Mao, Y., and Li, B. (2018). Second: Sparsely embedded convolutional detection. Sensors, 18.","DOI":"10.3390\/s18103337"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Lang, A.H., Vora, S., Caesar, H., Zhou, L., Yang, J., and Beijbom, O. (2019, January 15\u201320). Pointpillars: Fast encoders for object detection from point clouds. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.01298"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Qian, T., Chen, J., Zhuo, L., Jiao, Y., and Jiang, Y.G. (2024, January 26\u201327). Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.","DOI":"10.1609\/aaai.v38i5.28253"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"5668","DOI":"10.1109\/TITS.2021.3055616","article-title":"3D multi-object tracking in point clouds based on prediction confidence-guided data association","volume":"23","author":"Wu","year":"2021","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Chen, X., Ma, H., Wan, J., Li, B., and Xia, T. (2017, January 21\u201326). Multi-view 3d object detection network for autonomous driving. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.691"},{"key":"ref_26","unstructured":"Zhou, Y., Sun, P., Zhang, Y., Anguelov, D., Gao, J., Ouyang, T., Guo, J., Ngiam, J., and Vasudevan, V. (2020, January 16\u201318). End-to-end multi-view fusion for 3d object detection in lidar point clouds. Proceedings of the Conference on Robot Learning, Virtual."},{"key":"ref_27","first-page":"10421","article-title":"Bevfusion: A simple and robust lidar-camera fusion framework","volume":"35","author":"Liang","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"2020","DOI":"10.1109\/TPAMI.2024.3515454","article-title":"Bevformer: Learning bird\u2019s-eye-view representation from lidar-camera via spatiotemporal transformers","volume":"47","author":"Li","year":"2024","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Liu, Y., Wang, T., Zhang, X., and Sun, J. (2022, January 23\u201327). Petr: Position embedding transformation for multi-view 3d object detection. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.","DOI":"10.1007\/978-3-031-19812-0_31"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Ku, J., Mozifian, M., Lee, J., Harakeh, A., and Waslander, S.L. (2018, January 1\u20135). Joint 3d proposal generation and object detection from view aggregation. Proceedings of the 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.","DOI":"10.1109\/IROS.2018.8594049"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"3537","DOI":"10.1109\/TPAMI.2023.3346386","article-title":"3d object detection from images for autonomous driving: A survey","volume":"46","author":"Ma","year":"2023","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_32","unstructured":"Wang, H., Zhu, R., Wu, H., Ye, Z., Deng, S., and Li, B. (2025, January 20\u201321). Object detection and tracking based on image and point cloud instance segmentation matching for robot. Proceedings of the 2025 6th International Conference on Mechanical Engineering and Materials (ICMEM 2025), Nanchang, China. (Accepted)."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/2\/718\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,21]],"date-time":"2026-01-21T14:04:29Z","timestamp":1769004269000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/2\/718"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,21]]},"references-count":32,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2026,1]]}},"alternative-id":["s26020718"],"URL":"https:\/\/doi.org\/10.3390\/s26020718","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,21]]}}}