{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T22:09:17Z","timestamp":1740175757987,"version":"3.37.3"},"reference-count":49,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2023,6,29]],"date-time":"2023-06-29T00:00:00Z","timestamp":1687996800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,6,29]],"date-time":"2023-06-29T00:00:00Z","timestamp":1687996800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Complex Intell. Syst."],"published-print":{"date-parts":[[2023,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The external calibration between 3D LiDAR and 2D camera is an extremely important step towards multimodal fusion for robot perception. However, its accuracy is still unsatisfactory. To improve the accuracy of calibration, we first analyze the interference factors that affect the performance of the calibration model under a causal inference framework in this study. Guided by the causality analysis, we present Iter-CalibNet (Iterative Calibration Convolutional Neural Network) to infer a 6 degrees of freedom (DoF) rigid body transformation between 3D LiDAR and 2D camera. By downscaling point clouds to obtain more overlapping region between 3D\u20132D data pair and applying iterative calibration manner, the interference of confounding bias in the calibration model is effectively eliminated. Moreover, our Iter-CalibNet adds non-local neural network after each convolution operation to capture the transformation relationship. We also combine the geometric loss and photometric loss obtained from the interframe constraints to optimize the calibration accuracy. Extensive experiments demonstrate that our Iter-CalibNet can achieve leading performance by comparison with other CNN based and traditional calibration methods.<\/jats:p>","DOI":"10.1007\/s40747-023-01140-1","type":"journal-article","created":{"date-parts":[[2023,6,29]],"date-time":"2023-06-29T10:02:07Z","timestamp":1688032927000},"page":"7349-7363","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Causal calibration: iteratively calibrating LiDAR and camera by considering causality and geometry"],"prefix":"10.1007","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2130-9122","authenticated-orcid":false,"given":"Ruyu","family":"Liu","sequence":"first","affiliation":[]},{"given":"Jieying","family":"Shi","sequence":"additional","affiliation":[]},{"given":"Haoyu","family":"Zhang","sequence":"additional","affiliation":[]},{"given":"Jianhua","family":"Zhang","sequence":"additional","affiliation":[]},{"given":"Bo","family":"Sun","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,6,29]]},"reference":[{"issue":"12","key":"1140_CR1","doi-asserted-by":"publisher","first-page":"25180","DOI":"10.1109\/TITS.2022.3155925","volume":"23","author":"R Liu","year":"2022","unstructured":"Liu R, Zhang G, Wang J, Zhao S (2022) Cross-modal 360$$^\\circ $$ depth completion and reconstruction for large-scale indoor environment. IEEE Trans Intell Transp Syst 23(12):25180\u201325190","journal-title":"IEEE Trans Intell Transp Syst"},{"issue":"10","key":"1140_CR2","doi-asserted-by":"publisher","first-page":"618","DOI":"10.1007\/s11801-022-2045-2","volume":"18","author":"W Li","year":"2022","unstructured":"Li W, Zhang Z, Jiang Z, Gao X, Tan Z, Wang H (2022) A ransac based phase noise filtering method for the camera-projector calibration system. Optoelectron Lett 18(10):618\u2013622","journal-title":"Optoelectron Lett"},{"issue":"12","key":"1140_CR3","doi-asserted-by":"publisher","first-page":"730","DOI":"10.1007\/s11801-022-2101-y","volume":"18","author":"S Wang","year":"2022","unstructured":"Wang S, Qiao Z, Huang J (2022) A calibration method of external parameters of 2d laser in rotary 3d scanning. Optoelectron Lett 18(12):730\u2013736","journal-title":"Optoelectron Lett"},{"key":"1140_CR4","doi-asserted-by":"crossref","unstructured":"Wang Z, Zhang J, Chen S, Yuan C, Zhang J, Zhang J (2019) Robust high accuracy visual-inertial-laser slam system. In: 2019 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), 6636\u20136641. IEEE","DOI":"10.1109\/IROS40897.2019.8967702"},{"issue":"8","key":"1140_CR5","doi-asserted-by":"publisher","first-page":"1242","DOI":"10.1002\/rob.21809","volume":"35","author":"J Zhang","year":"2018","unstructured":"Zhang J, Singh S (2018) Laser-visual-inertial odometry and mapping with high robustness and low drift. J Field Robot 35(8):1242\u20131264","journal-title":"J Field Robot"},{"key":"1140_CR6","doi-asserted-by":"crossref","unstructured":"Zhou L, Li Z, Kaess M (2018) Automatic extrinsic calibration of a camera and a 3d lidar using line and plane correspondences. In: 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5562\u20135569 IEEE","DOI":"10.1109\/IROS.2018.8593660"},{"key":"1140_CR7","doi-asserted-by":"crossref","unstructured":"Liu H, Liu Y, Gu X, Wu Y, Qu F, Huang L (2018) A deep-learning based multi-modality sensor calibration method for usv. In: 2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM), pp. 1\u20135 . IEEE","DOI":"10.1109\/BigMM.2018.8499349"},{"key":"1140_CR8","unstructured":"Taylor ZJ (2015) Automatic markerless calibration of multi-modal sensor arrays. PhD thesis, University of Sydney"},{"key":"1140_CR9","doi-asserted-by":"crossref","unstructured":"Pusztai Z, Hajder L (2017) Accurate calibration of lidar-camera systems using ordinary boxes. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 394\u2013402","DOI":"10.1109\/ICCVW.2017.53"},{"key":"1140_CR10","doi-asserted-by":"crossref","unstructured":"Jiang, J, Xue P, Chen S, Liu Z, Zhang X, Zheng N (2018) Line feature based extrinsic calibration of lidar and camera. In: 2018 IEEE International Conference on Vehicular Electronics and Safety (ICVES), pp. 1\u20136 IEEE","DOI":"10.1109\/ICVES.2018.8519493"},{"key":"1140_CR11","doi-asserted-by":"crossref","unstructured":"Domhof J, FP KJ (2019) An extrinsic calibration tool for radar, camera and lidar. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 8107\u20138113 IEEE","DOI":"10.1109\/ICRA.2019.8794186"},{"key":"1140_CR12","doi-asserted-by":"crossref","unstructured":"Castorena J, Kamilov US, Boufounos PT (2016) Autocalibration of lidar and optical cameras via edge alignment. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2862\u20132866 IEEE","DOI":"10.1109\/ICASSP.2016.7472200"},{"key":"1140_CR13","unstructured":"Chien HJ, Klette R, Schneider N, Franke U (2016) Visual odometry driven online calibration for monocular lidar-camera systems. In: 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 2848\u20132853 . IEEE"},{"key":"1140_CR14","doi-asserted-by":"crossref","unstructured":"Schneider N, Piewak F, Stiller C, Franke U Regnet: Multimodal sensor registration using deep neural networks. In: 2017 IEEE Intelligent Vehicles Symposium (IV), pp. 1803\u20131810 (2017). IEEE","DOI":"10.1109\/IVS.2017.7995968"},{"key":"1140_CR15","doi-asserted-by":"crossref","unstructured":"Iyer G, Ram RK, Murthy JK, Krishna KM (2018) Calibnet: geometrically supervised extrinsic calibration using 3d spatial transformer networks. In: 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1110\u20131117 IEEE","DOI":"10.1109\/IROS.2018.8593693"},{"key":"1140_CR16","doi-asserted-by":"crossref","unstructured":"Shi J, Zhu Z, Zhang J, Liu R, Wang Z, Chen S, Liu H (2020) Calibrcnn: calibrating camera and lidar by recurrent convolutional neural network and geometric constraints. In: 2020 IEEE International Conference on Intelligent Robots and Systems (IROS)","DOI":"10.1109\/IROS45743.2020.9341147"},{"issue":"4","key":"1140_CR17","doi-asserted-by":"publisher","first-page":"6956","DOI":"10.1109\/LRA.2020.3026958","volume":"5","author":"K Yuan","year":"2020","unstructured":"Yuan K, Guo Z, Wang ZJ (2020) Rggnet: tolerance aware lidar\u2013camera online calibration with geometric deep learning and generative model. IEEE Robot Autom Lett 5(4):6956\u20136963. https:\/\/doi.org\/10.1109\/LRA.2020.3026958","journal-title":"IEEE Robot Autom Lett"},{"key":"1140_CR18","volume-title":"Causal Inference in statistics: a primer","author":"J Pearl","year":"2019","unstructured":"Pearl J, Glymour M, Jewell N (2019) Causal Inference in statistics: a primer. Wiley, Chichester"},{"issue":"11","key":"1140_CR19","doi-asserted-by":"publisher","first-page":"1231","DOI":"10.1177\/0278364913491297","volume":"32","author":"A Geiger","year":"2013","unstructured":"Geiger A, Lenz P, Stiller C, Urtasun R (2013) Vision meets robotics: the kitti dataset. Int J Robot Res 32(11):1231\u20131237","journal-title":"Int J Robot Res"},{"issue":"5","key":"1140_CR20","doi-asserted-by":"publisher","first-page":"1104","DOI":"10.1109\/TRO.2012.2200990","volume":"28","author":"M Bosse","year":"2012","unstructured":"Bosse M, Zlot R, Flick P (2012) Zebedee: design of a spring-mounted 3-d range sensor with application to mobile mapping. IEEE Trans Robot 28(5):1104\u20131119","journal-title":"IEEE Trans Robot"},{"key":"1140_CR21","doi-asserted-by":"crossref","unstructured":"Zhang J, Singh S (2014) Loam: lidar odometry and mapping in real-time. Robot Sci Syst 2","DOI":"10.15607\/RSS.2014.X.007"},{"key":"1140_CR22","doi-asserted-by":"crossref","unstructured":"Zhang J, Singh S (2015) Visual-lidar odometry and mapping: low-drift, robust, and fast. In: 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 2174\u20132181 IEEE","DOI":"10.1109\/ICRA.2015.7139486"},{"key":"1140_CR23","doi-asserted-by":"crossref","unstructured":"Kelly J, Sukhatme GS (2009) Fast relative pose calibration for visual and inertial sensors. In: Experimental Robotics, pp. 515\u2013524 Springer","DOI":"10.1007\/978-3-642-00196-3_59"},{"key":"1140_CR24","doi-asserted-by":"crossref","unstructured":"Furgal, P, Rehder J, Siegwart R. Unified temporal and spatial calibration for multi-sensor systems. In: 2013 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1280\u20131286 (2013). IEEE","DOI":"10.1109\/IROS.2013.6696514"},{"key":"1140_CR25","doi-asserted-by":"crossref","unstructured":"Dong-Si TC, Mourikis AI (2012) Estimator initialization in vision-aided inertial navigation with unknown camera-imu calibration. In: 2012 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1064\u20131071 IEEE","DOI":"10.1109\/IROS.2012.6386235"},{"issue":"1","key":"1140_CR26","doi-asserted-by":"publisher","first-page":"39","DOI":"10.1109\/TASE.2016.2550621","volume":"14","author":"Z Yang","year":"2016","unstructured":"Yang Z, Shen S (2016) Monocular visual-inertial state estimation with online initialization and camera-imu extrinsic calibration. IEEE Trans Automat Sci Eng 14(1):39\u201351","journal-title":"IEEE Trans Automat Sci Eng"},{"issue":"16","key":"1140_CR27","doi-asserted-by":"publisher","first-page":"336","DOI":"10.3182\/20100906-3-IT-2019.00059","volume":"43","author":"G Pandey","year":"2010","unstructured":"Pandey G, McBride J, Savarese S, Eustice R (2010) Extrinsic calibration of a 3d laser scanner and an omnidirectional camera. IFAC Proc Vol 43(16):336\u2013341","journal-title":"IFAC Proc Vol"},{"key":"1140_CR28","first-page":"7","volume":"2","author":"J Levinson","year":"2013","unstructured":"Levinson J, Thrun S (2013) Automatic online calibration of cameras and lasers. Roboti Sci Syst 2:7","journal-title":"Roboti Sci Syst"},{"issue":"5","key":"1140_CR29","doi-asserted-by":"publisher","first-page":"696","DOI":"10.1002\/rob.21542","volume":"32","author":"G Pandey","year":"2015","unstructured":"Pandey G, McBride JR, Savarese S, Eustice RM (2015) Automatic extrinsic calibration of vision and lidar by maximizing mutual information. J Field Robot 32(5):696\u2013722","journal-title":"J Field Robot"},{"key":"1140_CR30","doi-asserted-by":"crossref","unstructured":"Scott T, Morye AA, Pini\u00e9s P, Paz LM, Posner I, Newman P (2015) Exploiting known unknowns: Scene induced cross-calibration of lidar-stereo systems. In: 2015 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3647\u20133653 IEEE","DOI":"10.1109\/IROS.2015.7353887"},{"issue":"5","key":"1140_CR31","doi-asserted-by":"publisher","first-page":"1215","DOI":"10.1109\/TRO.2016.2596771","volume":"32","author":"Z Taylor","year":"2016","unstructured":"Taylor Z, Nieto J (2016) Motion-based calibration of multimodal sensor extrinsics and timing offset estimation. IEEE Trans Robot 32(5):1215\u20131229. https:\/\/doi.org\/10.1109\/TRO.2016.2596771","journal-title":"IEEE Trans Robot"},{"key":"1140_CR32","doi-asserted-by":"crossref","unstructured":"Engelcke M, Rao D, Wang DZ, Tong CH, Posner I (2017) Vote3deep: fast object detection in 3d point clouds using efficient convolutional neural networks. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 1355\u20131361 IEEE","DOI":"10.1109\/ICRA.2017.7989161"},{"key":"1140_CR33","doi-asserted-by":"crossref","unstructured":"Laina I, Rupprecht C, Belagiannis V, Tombari F, Navab N (2016) Deeper depth prediction with fully convolutional residual networks. In: 2016 Fourth International Conference on 3D Vision (3DV), pp. 239\u2013248 IEEE","DOI":"10.1109\/3DV.2016.32"},{"key":"1140_CR34","doi-asserted-by":"crossref","unstructured":"Zhou T, Brown M, Snavely N, Lowe DG (2017) Unsupervised learning of depth and ego-motion from video. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1851\u20131858","DOI":"10.1109\/CVPR.2017.700"},{"issue":"4","key":"1140_CR35","doi-asserted-by":"publisher","first-page":"1004","DOI":"10.1109\/TRO.2018.2853729","volume":"34","author":"T Qin","year":"2018","unstructured":"Qin T, Li P, Shen S (2018) Vins-mono: a robust and versatile monocular visual-inertial state estimator. IEEE Trans Robot 34(4):1004\u20131020","journal-title":"IEEE Trans Robot"},{"key":"1140_CR36","doi-asserted-by":"crossref","unstructured":"Wang S, Clark R, Wen H, Trigoni N (2017) Deepvo: Towards end-to-end visual odometry with deep recurrent convolutional neural networks. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 2043\u20132050 IEEE","DOI":"10.1109\/ICRA.2017.7989236"},{"key":"1140_CR37","doi-asserted-by":"crossref","unstructured":"Shen T, Luo Z, Zhou L, Deng H, Zhang R, Fang T, Quan L (2019) Beyond photometric loss for self-supervised ego-motion estimation. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 6359\u20136365 IEEE","DOI":"10.1109\/ICRA.2019.8793479"},{"key":"1140_CR38","doi-asserted-by":"crossref","unstructured":"Qi J, Niu Y, Huang J, Zhang H (2019) Two causal principles for improving visual dialog. CoRR .arXiv:1911.10496","DOI":"10.1109\/CVPR42600.2020.01087"},{"key":"1140_CR39","doi-asserted-by":"publisher","unstructured":"Tang K, Niu Y, Huang J, Shi J, Zhang H (2020) Unbiased scene graph generation from biased training. In: 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3713\u20133722. https:\/\/doi.org\/10.1109\/CVPR42600.2020.00377","DOI":"10.1109\/CVPR42600.2020.00377"},{"key":"1140_CR40","doi-asserted-by":"crossref","unstructured":"Zhang D, Zhang H, Tang J, Hua X, Sun Q (2020) Causal intervention for weakly-supervised semantic segmentation. In: 2020 Conference on Neural Information Processing Systems (NeurIPS)","DOI":"10.1109\/ICIP40778.2020.9190911"},{"key":"1140_CR41","unstructured":"Bareinboim E, Pearl J (2012) Controlling selection bias in causal inference. In: Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, pp. 100\u2013108"},{"key":"1140_CR42","unstructured":"Besserve M, Sun R, Sch\u00f6lkopf B (2018) Counterfactuals uncover the modular structure of deep generative models. CoRR. arXiv:1812.03253"},{"key":"1140_CR43","unstructured":"Parascandolo G, Rojas-Carulla M, Kilbertus N, Sch\u00f6lkopf B (2017) Learning independent causal mechanisms. CoRR. arXiv:1712.00961"},{"key":"1140_CR44","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770\u2013778","DOI":"10.1109\/CVPR.2016.90"},{"key":"1140_CR45","doi-asserted-by":"crossref","unstructured":"Lowe DG (1999) Object recognition from local scale-invariant features. In: Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 2, pp. 1150\u20131157 IEEE","DOI":"10.1109\/ICCV.1999.790410"},{"key":"1140_CR46","doi-asserted-by":"crossref","unstructured":"Zhou T, Tulsiani S, Sun W, Malik J, Efros AA (2016) View synthesis by appearance flow. In: European Conference on Computer Vision, pp. 286\u2013301 Springer","DOI":"10.1007\/978-3-319-46493-0_18"},{"key":"1140_CR47","unstructured":"Kingma D, Ba JA (2014) A method for stochastic optimization. In: International Conference on Learning Representations"},{"key":"1140_CR48","unstructured":"Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 249\u2013256"},{"key":"1140_CR49","doi-asserted-by":"crossref","unstructured":"Cattaneo D, Vaghi M, Ballardini AL, Fontana S, Sorrenti DG, Burgard W (2019) Cmrnet: camera to lidar-map registration. In: 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pp. 1283\u20131289 IEEE","DOI":"10.1109\/ITSC.2019.8917470"}],"container-title":["Complex &amp; Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-023-01140-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s40747-023-01140-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-023-01140-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,27]],"date-time":"2023-10-27T19:04:23Z","timestamp":1698433463000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s40747-023-01140-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,29]]},"references-count":49,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2023,12]]}},"alternative-id":["1140"],"URL":"https:\/\/doi.org\/10.1007\/s40747-023-01140-1","relation":{},"ISSN":["2199-4536","2198-6053"],"issn-type":[{"type":"print","value":"2199-4536"},{"type":"electronic","value":"2198-6053"}],"subject":[],"published":{"date-parts":[[2023,6,29]]},"assertion":[{"value":"6 December 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 May 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 June 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}