{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,19]],"date-time":"2026-03-19T07:03:37Z","timestamp":1773903817877,"version":"3.50.1"},"reference-count":49,"publisher":"MDPI AG","issue":"16","license":[{"start":{"date-parts":[[2021,8,19]],"date-time":"2021-08-19T00:00:00Z","timestamp":1629331200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61773082"],"award-info":[{"award-number":["61773082"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Research and Innovation Project for Postgraduate of Chongqing","award":["CYB18173"],"award-info":[{"award-number":["CYB18173"]}]},{"name":"Doctoral Innovative High-end Talents Project","award":["BYJS201808"],"award-info":[{"award-number":["BYJS201808"]}]},{"name":"State Scholarship Fund of China Scholarship Council","award":["201908500145"],"award-info":[{"award-number":["201908500145"]}]},{"name":"Chongqing Technology Innovation and application development special project","award":["cstc2020jscx-msxmX0153"],"award-info":[{"award-number":["cstc2020jscx-msxmX0153"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Since single sensor and high-density point cloud data processing have certain direct processing limitations in urban traffic scenarios, this paper proposes a 3D instance segmentation and object detection framework for urban transportation scenes based on the fusion of Lidar remote sensing technology and optical image sensing technology. Firstly, multi-source and multi-mode data pre-fusion and alignment of Lidar and camera sensor data are effectively carried out, and then a unique and innovative network of stereo regional proposal selective search-driven DAGNN is constructed. Finally, using the multi-dimensional information interaction, three-dimensional point clouds with multi-features and unique concave-convex geometric characteristics are instance over-segmented and clustered by the hypervoxel storage in the remarkable octree and growing voxels. Finally, the positioning and semantic information of significant 3D object detection in this paper are visualized by multi-dimensional mapping of the boundary box. The experimental results validate the effectiveness of the proposed framework with excellent feedback for small objects, object stacking, and object occlusion. It can be a remediable or alternative plan to a single sensor and provide an essential theoretical and application basis for remote sensing, autonomous driving, environment modeling, autonomous navigation, and path planning under the V2X intelligent network space\u2013 ground integration in the future.<\/jats:p>","DOI":"10.3390\/rs13163288","type":"journal-article","created":{"date-parts":[[2021,8,22]],"date-time":"2021-08-22T22:59:27Z","timestamp":1629673167000},"page":"3288","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":19,"title":["3D Instance Segmentation and Object Detection Framework Based on the Fusion of Lidar Remote Sensing and Optical Image Sensing"],"prefix":"10.3390","volume":"13","author":[{"given":"Ling","family":"Bai","sequence":"first","affiliation":[{"name":"Department of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yinguo","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Automation, Chongqing University of Posts and Telecommunications, Chongqing 400065, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ming","family":"Cen","sequence":"additional","affiliation":[{"name":"Department of Automation, Chongqing University of Posts and Telecommunications, Chongqing 400065, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fangchao","family":"Hu","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, Chongqing University of Technology, Chongqing 400054, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,8,19]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Pang, C., Zhong, X., Hu, H., Tian, J., Peng, X., and Zeng, J. (2018). Adaptive Obstacle Detection for Mobile Robots in Urban Environments Using Downward-Looking 2D LiDAR. Sensors, 18.","DOI":"10.3390\/s18061749"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"2270","DOI":"10.1109\/TPAMI.2014.2316828","article-title":"3D Object Recognition in Cluttered Scenes with Local Surface Features: A Survey","volume":"36","author":"Guo","year":"2014","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_3","unstructured":"Guo, Y., Wen, C., Sun, X., Wang, C., and Li, J. (August, January 28). Partial 3D Object Retrieval and Completeness Evaluation for Urban Street Scene. Proceedings of the IEEE International Geoscience and Remote Sensing Symposium, Yokohama, Japan."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Zou, K., Zhang, Z., Zhang, J., and Zhang, Q. (2015, January 15\u201317). A 3D model feature extraction method using curvature-based shape distribution. Proceedings of the IEEE International Conference on Fuzzy Systems & Knowledge Discovery, Zhangjiajie, China.","DOI":"10.1109\/FSKD.2015.7382221"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"118","DOI":"10.1006\/cviu.2001.0919","article-title":"Curvature Computation on Free-Form 3-D Meshes at Multiple Scales","volume":"83","author":"Mokhtarian","year":"2001","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"543","DOI":"10.1109\/JSTSP.2010.2096797","article-title":"A New Weighted Fuzzy C-Means Clustering Algorithm for Remotely Sensed Image Classification","volume":"5","author":"Hung","year":"2010","journal-title":"IEEE J. Sel. Top. Signal Process."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1258","DOI":"10.1109\/TPAMI.2015.2477823","article-title":"Scale Space Graph Representation and Kernel Matching for Non-Rigid and Textured 3D Shape Retrieval","volume":"38","author":"Garro","year":"2015","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Chen, B., Chen, H., Yuan, D., and Yu, L. (2020). 3D Fast Object Detection Based on Discriminant Images and Dynamic Distance Threshold Clustering. Sensors, 20.","DOI":"10.3390\/s20247221"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1051","DOI":"10.1016\/j.dsp.2020.102818","article-title":"Three-branch architecture for stereoscopic 3D salient object detection","volume":"106","author":"Zhou","year":"2020","journal-title":"Digital Signal Process."},{"key":"ref_10","unstructured":"Luo, Q., Ma, H., Tang, L., Wang, Y., and Xiong, R. (2017). 3D-SSD: Learning hierarchical features from RGB-D images for amodal 3D object detection. arXiv."},{"key":"ref_11","first-page":"6009","article-title":"A Bayesian Filter for Multi-view 3D Multi-object Tracking with Occlusion Handling","volume":"12","author":"Ong","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Awadallah, M., Abbott, L., and Ghannam, S. (2014, January 27\u201330). Segmentation of sparse noisy point clouds using active contour models. Proceedings of the IEEE International Conference on Image Processing (ICIP), Paris, France.","DOI":"10.1109\/ICIP.2014.7026223"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"274","DOI":"10.1007\/978-3-662-45737-5_28","article-title":"A Segmentation Method for Point Cloud Based on Local Sample and Statistic Inference","volume":"482","author":"Wang","year":"2015","journal-title":"Geoinform. Resour. Manag. Sustain. Ecosyst."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Li, L., Yang, F., Zhu, H., Li, D., Li, Y., and Tang, L. (2017). An Improved RANSAC for 3D Point Cloud Plane Segmentation Based on Normal Distribution Transformation Cells. Remote Sens., 9.","DOI":"10.3390\/rs9050433"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"377","DOI":"10.1080\/2150704X.2020.1847348","article-title":"A new approach for roof segmentation from airborne LiDAR point clouds","volume":"12","author":"Zhao","year":"2021","journal-title":"Remote Sens. Lett."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Xu, B., Jiang, W., Shan, J., Zhang, J., and Li, L. (2016). Investigation on the Weighted RANSAC Approaches for Building Roof Plane Segmentation from LiDAR Point Clouds. Remote Sens., 8.","DOI":"10.3390\/rs8010005"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1693","DOI":"10.1109\/TVCG.2018.2827998","article-title":"Multi-Normal Estimation via Pair Consistency Voting","volume":"25","author":"Zhang","year":"2018","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Dey, E., Kurdi, F.T., Awrangjeb, M., and Stantic, B. (2021). Effective Selection of Variable Point Neighbourhood for Feature Point Extraction from Aerial Building Point Cloud Data. Remote Sens., 13.","DOI":"10.3390\/rs13081520"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"107443","DOI":"10.1016\/j.patcog.2020.107443","article-title":"Cylinders extraction in non-oriented point clouds as a clustering problem","volume":"107","author":"Bergamasco","year":"2020","journal-title":"Pattern Recognit."},{"key":"ref_20","first-page":"306","article-title":"Real-Time Plane Segmentation Using RGB-D Cameras","volume":"Volume 7416","author":"Dirk","year":"2011","journal-title":"Robot Soccer World Cup"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Hu, F., Tian, Z., Li, Y., Huang, S., and Feng, M. (2018, January 9\u201311). A Combined Clustering and Image Mapping based Point Cloud Segmentation for 3D Object Detection. Proceedings of the Chinese Control and Decision Conference, Shenyang, China.","DOI":"10.1109\/CCDC.2018.8407395"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"910","DOI":"10.1109\/LGRS.2020.2987747","article-title":"Boundary-Aware and Semiautomatic Segmentation of 3-D Object in Point Clouds","volume":"18","author":"Luo","year":"2021","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1016\/j.isprsjprs.2015.01.011","article-title":"Octree-based region growing for point cloud segmentation","volume":"104","author":"Vo","year":"2015","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Li, L., Yao, J., Tu, J., Liu, X., Li, Y., and Guo, L. (2020). Roof Plane Segmentation from Airborne LiDAR Data Using Hierarchical Clustering and Boundary Relabeling. Remote Sens., 12.","DOI":"10.3390\/rs12091363"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Hasirci, Z., and Ozturk, M. (2015, January 9\u201311). The comparison of region growing algorithms with using EMST for point clouds. Proceedings of the International Conference on Telecommunications and Signal Processing (TSP), Prague, Czech Republic.","DOI":"10.1109\/TSP.2015.7296396"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"4160","DOI":"10.1109\/JSTARS.2019.2936662","article-title":"An Accurate and Robust Region-Growing Algorithm for Plane Segmentation of TLS Point Clouds Using a Multiscale Tensor Voting Method","volume":"12","author":"Wu","year":"2019","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Strom, J., Richardson, A., and Olson, E. (2010, January 18\u201322). Graph-based segmentation for colored 3D laser point clouds. Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems, Taipei, Taiwan.","DOI":"10.1109\/IROS.2010.5650459"},{"key":"ref_28","unstructured":"Tatavarti, A., Papadakis, J., and Willis, A.R. (April, January 30). Towards real-time segmentation of 3D point cloud data into local planar regions. Proceedings of the SoutheastCon, Concord, NC, USA."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1655","DOI":"10.1109\/LSP.2020.3023587","article-title":"Hypergraph Spectral Clustering for Point Cloud Segmentation","volume":"27","author":"Zhang","year":"2020","journal-title":"IEEE Signal Process. Lett."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1016\/j.isprsjprs.2020.11.016","article-title":"Combining graph-cut clustering with object-based stem detection for tree segmentation in highly dense airborne lidar point clouds","volume":"172","author":"Sd","year":"2021","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_31","unstructured":"Charles, R.Q., Wei, L., Chenxia, W., Hao, S., and Leonidas, J.G. (2018, January 18\u201323). Frustum PointNets for 3D Object Detection from RGB-D Data. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Lin, Z.-H., Huang, S.Y., and Wang, Y.-C.F. (2021). Learning of 3D Graph Convolution Networks for Point Cloud Analysis. IEEE Trans. Pattern Anal. Mach. Intell.","DOI":"10.1109\/TPAMI.2021.3059758"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"300","DOI":"10.1016\/j.neucom.2020.12.067","article-title":"Geometric attentional dynamic graph convolutional neural networks for point cloud analysis","volume":"432","author":"Cui","year":"2021","journal-title":"Neurocomputing"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Wang, J., Xu, C., Dai, L., Zhang, J., and Zhong, R.Y. (2021). An Unequal Learning Approach for 3D Point Cloud Segmentation. IEEE Trans. Ind. Inform.","DOI":"10.1109\/TII.2020.3044106"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Nagy, B., and Benedek, C. (2020). On-the-Fly Camera and Lidar Calibration. Remote Sens., 12.","DOI":"10.3390\/rs12071137"},{"key":"ref_36","unstructured":"Geiger, A., Moosmann, F., Car, O., and Schuster, B. (2012, January 14\u201318). A toolbox for automatic calibration of range and camera sensors using a single shot. Proceedings of the International Conference on Robotics and Automation (ICRA), Saint Paul, MN, USA."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Bai, L., Li, Y., Hu, F., and Zhao, F. (2019, January 3\u20135). Region-proposal Convolutional Network-driven Point Cloud Voxelization and Over-segmentation for 3D Object Detection. Proceedings of the IEEE Chinese Control and Decision Conference (CCDC), Nanchang, China.","DOI":"10.1109\/CCDC.2019.8832629"},{"key":"ref_38","first-page":"155","article-title":"Color-Based Segmentation of Point Clouds","volume":"38","author":"Qingming","year":"2009","journal-title":"Laser Scanning"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Stein, S.C., Schoeler, M., Papon, J., and Worgotter, F. (2014, January 23\u201328). Object partitioning using local convexity. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.46"},{"key":"ref_40","unstructured":"Golovinskiy, A., and Funkhouser, T. (October, January 27). Min-cut based segmentation of point clouds. Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCV), Kyoto, Japan."},{"key":"ref_41","first-page":"248","article-title":"Segmentation of point clouds using smoothness constraint","volume":"36","author":"Rabbani","year":"2006","journal-title":"Int. Arch. Photogramm. Remote. Sens. Spat. Inf. Sci."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Papon, J., Abramov, A., Schoeler, M., and Worgotter, F. (2013, January 23\u201328). Voxel Cloud Connectivity Segmentation\u2014Supervoxels for Point Clouds. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.","DOI":"10.1109\/CVPR.2013.264"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"David, C., Nafornita, C., Gui, V., Campeanu, A., Carrie, G., and Monnerat, M. (2021). GNSS Localization in Constraint Environment by Image Fusing Techniques. Remote Sens., 13.","DOI":"10.3390\/rs13102021"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Beltran, J., Guindel, C., Moreno, F.M., Cruzado, D., Garcia, F., and De La Escalera, A. (2018, January 4\u20137). BirdNet: A 3D Object Detection Framework from LiDAR Information. Proceedings of the IEEE 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA.","DOI":"10.1109\/ITSC.2018.8569311"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Reading, C., Harakeh, A., Chae, J., and Waslander, S.L. (2021, January 19\u201325). Categorical depth distribution network for monocular 3d object detection. Proceedings of the International Conference on Computer Vision and Pattern Recognition, Paris, France.","DOI":"10.1109\/CVPR46437.2021.00845"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Shi, X., Ye, Q., Chen, X., Chen, C., Chen, Z., and Kim, T.K. (2021). Geometry-based distance decomposition for monocular 3D object de-tection. arXiv.","DOI":"10.1109\/ICCV48922.2021.01489"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Liu, Y., Wang, L., and Liu, M. (2021). Yolostereo3D: A step back to 2D for efficient stereo 3D detection. arXiv.","DOI":"10.1109\/ICRA48506.2021.9561423"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"3434","DOI":"10.1109\/LRA.2018.2852843","article-title":"RT3D: Real-Time 3-D Vehicle Detection in LiDAR Point Cloud for Autonomous Driving","volume":"3","author":"Zeng","year":"2018","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_49","unstructured":"Hu, H.N., Yang, Y.H., Fischer, T., Darrell, T., Yu, F., and Sin, M. (2021). Monocular quasi-dense 3D object tracking. arXiv."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/13\/16\/3288\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:47:36Z","timestamp":1760165256000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/13\/16\/3288"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,19]]},"references-count":49,"journal-issue":{"issue":"16","published-online":{"date-parts":[[2021,8]]}},"alternative-id":["rs13163288"],"URL":"https:\/\/doi.org\/10.3390\/rs13163288","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,8,19]]}}}