{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T13:53:01Z","timestamp":1782309181866,"version":"3.54.5"},"reference-count":41,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,4,18]],"date-time":"2023-04-18T00:00:00Z","timestamp":1681776000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:p>The integration of multiple sensors is a crucial and emerging trend in the development of autonomous driving technology. The depth image obtained by stereo matching of the binocular camera is easily influenced by environment and distance. The point cloud of LiDAR has strong penetrability. However, it is much sparser than binocular images. LiDAR-stereo fusion can neutralize the advantages of the two sensors and maximize the acquisition of reliable three-dimensional information to improve the safety of automatic driving. Cross-sensor fusion is a key issue in the development of autonomous driving technology. This study proposed a real-time LiDAR-stereo depth completion network without 3D convolution to fuse point clouds and binocular images using injection guidance. At the same time, a kernel-connected spatial propagation network was utilized to refine the depth. The output of dense 3D information is more accurate for autonomous driving. Experimental results on the KITTI dataset showed that our method used real-time techniques effectively. Further, we demonstrated our solution's ability to address sensor defects and challenging environmental conditions using the p-KITTI dataset.<\/jats:p>","DOI":"10.3389\/fnbot.2023.1124676","type":"journal-article","created":{"date-parts":[[2023,4,18]],"date-time":"2023-04-18T04:35:18Z","timestamp":1681792518000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Real-time depth completion based on LiDAR-stereo for autonomous driving"],"prefix":"10.3389","volume":"17","author":[{"given":"Ming","family":"Wei","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ming","family":"Zhu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yaoyuan","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiarong","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiaqi","family":"Sun","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2023,4,18]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"405","DOI":"10.1109\/3DIMPVT.2011.58","article-title":"Integrating LIDAR into Stereo for Fast and Improved Disparity Computation","author":"Badino","year":"2011","journal-title":"2011 International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission."},{"key":"B2","first-page":"11208","article-title":"Estimating Depth from RGB and Sparse Sensing","volume-title":"Lecture Notes in Computer Science.","author":"Chen","year":"2018"},{"key":"B3","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i07.6635","article-title":"CSPN++: learning context and resource aware convolutional spatial propagation networks for depth completion","author":"Cheng","year":"","journal-title":"ArXiv"},{"key":"B4","doi-asserted-by":"publisher","first-page":"2361","DOI":"10.1109\/TPAMI.2019.2947374","article-title":"Learning depth with convolutional spatial propagation network","author":"Cheng","year":"2018","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"B5","first-page":"6332","article-title":"Noise-Aware Unsupervised Deep Lidar-Stereo Fusion","volume-title":"2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Cheng","year":""},{"key":"B6","doi-asserted-by":"publisher","first-page":"4672","DOI":"10.1109\/LRA.2021.3068712","article-title":"Volumetric propagation network: stereo-LiDAR fusion for long-range depth estimation","author":"Choe","year":"2021","journal-title":"IEEE Robotics and Automation Letters"},{"key":"B7","doi-asserted-by":"crossref","first-page":"13641","DOI":"10.1109\/ICRA48506.2021.9561557","article-title":"Stereo-augmented Depth Completion from a Single RGB-LiDAR image","volume-title":"2021 IEEE International Conference on Robotics and Automation (ICRA)","author":"Choi","year":"2021"},{"key":"B8","doi-asserted-by":"crossref","first-page":"1","DOI":"10.23919\/AEITAUTOMOTIVE50086.2020.9307398","article-title":"LiDAR-stereo camera fusion for accurate depth estimation","volume-title":"2020 AEIT International Conference of Electrical and Electronic Technologies for Automotive (AEIT AUTOMOTIVE)","author":"Cholakkal","year":"2020"},{"key":"B9","doi-asserted-by":"publisher","first-page":"722","DOI":"10.1109\/TITS.2020.3023541","article-title":"Deep learning for image and point cloud fusion in autonomous driving: a review","volume":"23","author":"Cui","year":"2022","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell"},{"key":"B10","doi-asserted-by":"publisher","first-page":"2423","DOI":"10.1109\/TPAMI.2019.2929170","article-title":"Confidence propagation through cnns for guided sparse depth regression","volume":"42","author":"Eldesokey","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell"},{"key":"B11","doi-asserted-by":"publisher","first-page":"1995","DOI":"10.1093\/jcde\/qwac088","article-title":"Mode confusion of human-machine interfaces for automated vehicles","volume":"9","author":"Eom","year":"2022","journal-title":"J Comput Des Eng"},{"key":"B12","first-page":"11","volume-title":"Vision Meets Robotics: the Kitti Dataset.","author":"Geiger","year":"2013"},{"key":"B13","doi-asserted-by":"crossref","first-page":"13656","DOI":"10.1109\/ICRA48506.2021.9561035","article-title":"PENet: Towards Precise and Efficient Image Guided Depth Completion","volume-title":"2021 IEEE International Conference on Robotics and Automation (ICRA)","author":"Hu","year":"2021"},{"key":"B14","doi-asserted-by":"publisher","first-page":"3429","DOI":"10.1109\/TIP.2019.2960589","article-title":"HMS-Net: hierarchical multi-scale sparsity-invariant network for sparse depth completion","volume":"29","author":"Huang","year":"2020","journal-title":"IEEE Trans. Image Process."},{"key":"B15","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1109\/3DV.2018.00017","article-title":"Sparse and Dense Data with CNNs: Depth Completion and Semantic Segmentation","volume-title":"2018 International Conference on 3D Vision (3DV)","author":"Jaritz","year":"2018"},{"key":"B16","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.17","article-title":"End-to-end learning of geometry and context for deep stereo regression","author":"Kendall","year":"2017","journal-title":"ArXiv"},{"key":"B17","doi-asserted-by":"publisher","first-page":"1397","DOI":"10.1093\/jcde\/qwac061","article-title":"Safety monitoring system of personal mobility driving using deep learning","volume":"9","author":"Kim","year":"2022","journal-title":"J. Comput. Des. Eng."},{"key":"B18","doi-asserted-by":"crossref","first-page":"3288","DOI":"10.1109\/ICRA.2019.8793637","article-title":"Self-supervised sparse-to-dense: self-supervised depth completion from LiDAR and monocular camera","volume-title":"2019 International Conference on Robotics and Automation (ICRA)","author":"Ma","year":"2019"},{"key":"B19","doi-asserted-by":"crossref","first-page":"2181","DOI":"10.1109\/IROS.2016.7759342","article-title":"Real-time probabilistic fusion of sparse 3D LiDAR and dense stereo","volume-title":"2016 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)","author":"Maddern","year":"2016"},{"key":"B20","doi-asserted-by":"publisher","first-page":"150","DOI":"10.1049\/icp.2021.1442","article-title":"Sparse LiDAR and stereo fusion (SLS-Fusion) for depth estimation and 3D object detection","author":"Mai","year":"2021","journal-title":"11th International Conference of Pattern Recognition Systems (ICPRS 2021)"},{"key":"B21","doi-asserted-by":"crossref","first-page":"523","DOI":"10.23919\/DATE51398.2021.9474009","article-title":"A GPU-accelerated deep stereo-LiDAR fusion for real-time high-precision dense depth sensing","volume-title":"2021 Design, Automation and Test in Europe Conference and Exhibition (DATE)","author":"Meng","year":"2021"},{"key":"B22","doi-asserted-by":"publisher","DOI":"10.1109\/WACV48630.2021.00157","article-title":"Centerfusion: center-based radar and camera fusion for 3d object detection","author":"Nabati","year":"2020","journal-title":"ArXiv"},{"key":"B23","first-page":"65","article-title":"Fusion of Lidar and stereo range for mobile robots","volume-title":"International Conference on Advanced Robotics (ICAR).","author":"Nickels","year":"2003"},{"key":"B24","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-030-58601-0_8","article-title":"Non-Local Spatial Propagation Network for Depth Completion","volume-title":"European Conference on Computer Vision 12358","author":"Park","year":"2020"},{"key":"B25","doi-asserted-by":"publisher","first-page":"321","DOI":"10.1109\/TITS.2019.2891788","article-title":"High-precision depth estimation using uncalibrated LiDAR and stereo fusion","volume":"21","author":"Park","year":"2020","journal-title":"IEEE Trans. Intell Transp. Syst."},{"key":"B26","doi-asserted-by":"publisher","first-page":"3308","DOI":"10.1109\/CVPR.2019.00343","article-title":"DeepLiDAR: deep surface normal guided depth prediction for outdoor scene from sparse LiDAR data and single color image","author":"Qiu","year":"2019","journal-title":"2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)"},{"key":"B27","doi-asserted-by":"publisher","first-page":"8006","DOI":"10.1109\/TCYB.2020.3026425","article-title":"An \u03b1-\u03b2-divergence-generalized recommender for highly accurate predictions of missing user preferences","volume":"52","author":"Shang","year":"2021","journal-title":"IEEE Trans. Cybern"},{"key":"B28","doi-asserted-by":"publisher","first-page":"14357","DOI":"10.1109\/CVPR46437.2021.01413","article-title":"HITNet: hierarchical iterative tile refinement network for real-time stereo matching","author":"Tankovich","year":"2021","journal-title":"2021 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)"},{"key":"B29","first-page":"4489","article-title":"Learning spatiotemporal features with 3d convolutional networks","volume-title":"2015 IEEE International Conference on Computer Vision (ICCV)","author":"Tran","year":"2014"},{"key":"B30","first-page":"1","article-title":"Sparse and Noisy LiDAR Completion with RGB Guidance and Uncertainty","volume-title":"International Conference on Machine Vision Applications (MVA)","author":"Van Gansbeke","year":"2019"},{"key":"B31","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2011.00652","article-title":"Multi-view adaptive fusion network for 3d object detection","author":"Wang","year":"2020","journal-title":"ArXiv"},{"key":"B32","doi-asserted-by":"publisher","first-page":"5895","DOI":"10.1109\/IROS40897.2019.8968170","article-title":"3D LiDAR and Stereo Fusion using Stereo Matching Network with Conditional Cost Volume Normalization","author":"Wang","year":"2019","journal-title":"2019 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)"},{"key":"B33","doi-asserted-by":"publisher","first-page":"4689","DOI":"10.3390\/rs14194689","article-title":"An efficient information-reinforced lidar deep completion network without rgb guided","volume":"14","author":"Wei","year":"2022","journal-title":"Remote Sens."},{"key":"B34","doi-asserted-by":"publisher","first-page":"1956","DOI":"10.1109\/CVPR42600.2020.00203","article-title":"AANet: Adaptive Aggregation Network for Efficient Stereo Matching","author":"Xu","year":"2020","journal-title":"2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)"},{"key":"B35","doi-asserted-by":"publisher","first-page":"3348","DOI":"10.1109\/CVPR.2019.00347","article-title":"Dense depth posterior (DDP) from single image and sparse range","author":"Yang","year":"2019","journal-title":"2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)"},{"key":"B36","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1906.06310","article-title":"Pseudo-LiDAR++: accurate depth for 3D object detection in autonomous driving. International Conference on Learning Representations","author":"You","year":"2020","journal-title":"ArXiv"},{"key":"B37","doi-asserted-by":"publisher","DOI":"10.1109\/TCYB.2022.3185117","article-title":"A kalman-filter-incorporated latent factor analysis model for temporally dynamic sparse data","author":"Yuan","year":"2022","journal-title":"IEEE Trans. Cybern."},{"key":"B38","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1109\/CVPR.2019.00027","article-title":"GA-Net: guided aggregation net for end-to-end stereo matching","volume-title":"2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Zhang","year":"2019"},{"key":"B39","doi-asserted-by":"crossref","first-page":"7829","DOI":"10.1109\/ICRA40945.2020.9196628","article-title":"LiStereo: generate dense depth maps from LIDAR and stereo imagery","volume-title":"2020 IEEE International Conference on Robotics and Automation (ICRA)","author":"Zhang","year":"2020"},{"key":"B40","doi-asserted-by":"publisher","first-page":"10605","DOI":"10.1109\/LRA.2022.3191235","article-title":"SLFNet: a stereo and LiDAR fusion network for depth completion","volume":"7","author":"Zhang","year":"2022","journal-title":"IEEE Robotics Auto. Lett"},{"key":"B41","doi-asserted-by":"publisher","first-page":"5264","DOI":"10.1109\/TIP.2021.3079821","article-title":"Adaptive context-aware multi-modal network for depth completion","volume":"30","author":"Zhao","year":"2021","journal-title":"IEEE Trans. Image Proc"}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1124676\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,4,18]],"date-time":"2023-04-18T04:35:28Z","timestamp":1681792528000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1124676\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,18]]},"references-count":41,"alternative-id":["10.3389\/fnbot.2023.1124676"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2023.1124676","relation":{},"ISSN":["1662-5218"],"issn-type":[{"value":"1662-5218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,4,18]]},"article-number":"1124676"}}