{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,14]],"date-time":"2026-03-14T10:41:24Z","timestamp":1773484884414,"version":"3.50.1"},"reference-count":36,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2020,4,2]],"date-time":"2020-04-02T00:00:00Z","timestamp":1585785600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Sensor fusion is one of the main challenges in self driving and robotics applications. In this paper we propose an automatic, online and target-less camera-Lidar extrinsic calibration approach. We adopt a structure from motion (SfM) method to generate 3D point clouds from the camera data which can be matched to the Lidar point clouds; thus, we address the extrinsic calibration problem as a registration task in the 3D domain. The core step of the approach is a two-stage transformation estimation: First, we introduce an object level coarse alignment algorithm operating in the Hough space to transform the SfM-based and the Lidar point clouds into a common coordinate system. Thereafter, we apply a control point based nonrigid transformation refinement step to register the point clouds more precisely. Finally, we calculate the correspondences between the 3D Lidar points and the pixels in the 2D camera domain. We evaluated the method in various real-life traffic scenarios in Budapest, Hungary. The results show that our proposed extrinsic calibration approach is able to provide accurate and robust parameter settings on-the-fly.<\/jats:p>","DOI":"10.3390\/rs12071137","type":"journal-article","created":{"date-parts":[[2020,4,2]],"date-time":"2020-04-02T13:39:34Z","timestamp":1585834774000},"page":"1137","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":21,"title":["On-the-Fly Camera and Lidar Calibration"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0338-863X","authenticated-orcid":false,"given":"Bal\u00e1zs","family":"Nagy","sequence":"first","affiliation":[{"name":"Machine Perception Research Laboratory, Institute for Computer Science and Control, Kende Str. 13-17, 1111 Budapest, Hungary"},{"name":"3in-PPCU Research Group, Faculty of Information Technology and Bionics, P\u00e1zm\u00e1ny P\u00e9ter Catholic University, 2500 Esztergom, Hungary"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3203-0741","authenticated-orcid":false,"given":"Csaba","family":"Benedek","sequence":"additional","affiliation":[{"name":"Machine Perception Research Laboratory, Institute for Computer Science and Control, Kende Str. 13-17, 1111 Budapest, Hungary"},{"name":"3in-PPCU Research Group, Faculty of Information Technology and Bionics, P\u00e1zm\u00e1ny P\u00e9ter Catholic University, 2500 Esztergom, Hungary"},{"name":"Faculty of Informatics, University of Debrecen, Kassai \u00fat 26, 4028 Debrecen, Hungary"}]}],"member":"1968","published-online":{"date-parts":[[2020,4,2]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"696","DOI":"10.1002\/rob.21542","article-title":"Automatic Extrinsic Calibration of Vision and Lidar by Maximizing Mutual Information","volume":"32","author":"Pandey","year":"2015","journal-title":"J. Field Robot."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Pusztai, Z., Eichhardt, I., and Hajder, L. (2018). Accurate Calibration of Multi-LiDAR-Multi-Camera Systems. Sensors, 18.","DOI":"10.3390\/s18072139"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Iyer, G., Ram, R.K., Murthy, J.K., and Krishna, K.M. (2018, January 1\u20135). CalibNet: Geometrically Supervised Extrinsic Calibration using 3D Spatial Transformer Networks. Proceedings of the 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.","DOI":"10.1109\/IROS.2018.8593693"},{"key":"ref_4","unstructured":"Scaramuzza, D., Harati, A., and Siegwart, R. (November, January 29). Extrinsic self calibration of a camera and a 3D laser range finder from natural scenes. Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems, San Diego, CA, USA."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Moghadam, P., Bosse, M., and Zlot, R. (2013, January 6\u201310). Line-based extrinsic calibration of range and image sensors. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Karlsruhe, Germany.","DOI":"10.1109\/ICRA.2013.6631095"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Nagy, B., Kov\u00e1cs, L., and Benedek, C. (2019, January 27\u201331). Online Targetless End-to-End Camera-LIDAR Self-calibration. Proceedings of the International Conference on Machine Vision Applications (MVA), Tokyo, Japan.","DOI":"10.23919\/MVA.2019.8757887"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Nagy, B., Kov\u00e1cs, L., and Benedek, C. (2019, January 22\u201329). SFM and Semantic Information Based Online Targetless Camera-Lidar Self-Calibration. Proceedings of the IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan.","DOI":"10.1109\/ICIP.2019.8804299"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Geiger, A., Moosmann, F., Car, O., and Schuster, B. (2012, January 14\u201318). Automatic camera and range sensor calibration using a single shot. Proceedings of the IEEE International Conference on Robotics and Automation, St. Paul, MN, USA.","DOI":"10.1109\/ICRA.2012.6224570"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Alismail, H., Baker, L., and Browning, B. (2012, January 13\u201315). Automatic calibration of a range sensor and camera system. Proceedings of the 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission, Zurich, Switzerland.","DOI":"10.1109\/3DIMPVT.2012.52"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"5333","DOI":"10.3390\/s140305333","article-title":"Calibration between Color Camera and 3D LIDAR Instruments with a Polygonal Planar Board","volume":"14","author":"Park","year":"2014","journal-title":"Sensors"},{"key":"ref_11","unstructured":"Velas, M., Spanel, M., Materna, Z., and Herout, A. (2014). Calibration of RGB Camera With Velodyne LiDAR. WSCG 2014 Communication Papers Proceedings, Union Agency."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Rodriguez, F.S.A., Fremont, V., and Bonnifait, P. (2008, January 20\u201322). Extrinsic calibration between a multi-layer lidar and a camera. Proceedings of the IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, Seoul, Korea.","DOI":"10.1109\/MFI.2008.4648067"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1109\/70.88014","article-title":"Calibration of wrist-mounted robotic sensors by solving homogeneous transform equations of the form AXDXB","volume":"5","author":"Shiu","year":"1989","journal-title":"IEEE Trans. Robot. Autom."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Huang, K., and Stachniss, C. (2017, January 24\u201328). Extrinsic multi-sensor calibration for mobile robots using the Gauss Helmert model. Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada.","DOI":"10.1109\/IROS.2017.8205952"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Strobl, K.H., and Hirzinger, G. (2006, January 9\u201315). Optimal hand-eye calibration. Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Beijing, China.","DOI":"10.1109\/IROS.2006.282250"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"120106","DOI":"10.1109\/ACCESS.2019.2937909","article-title":"Extrinsic Calibration and Odometry for Camera-LiDAR Systems","volume":"7","author":"Shi","year":"2019","journal-title":"IEEE Access"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"J\u00fczsa, O., B\u00f6rcs, A., and Benedek, C. (2013, January 27\u201329). Towards 4D Virtual City Reconstruction From Lidar Point Cloud Sequences. Proceedings of the ISPRS Workshop on 3D Virtual City Modeling, Istanbul, Turkey.","DOI":"10.5194\/isprsannals-II-3-W1-15-2013"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Wang, R., Ferrie, F.P., and Macfarlane, J. (2012, January 16\u201321). Automatic registration of mobile LiDAR and spherical panoramas. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Providence, RI, USA.","DOI":"10.1109\/CVPRW.2012.6238912"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Napier, A., Corke, P., and Newman, P. (2013, January 6\u201310). Cross-calibration of push-broom 2D LIDARs and cameras in natural scenes. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Karlsruhe, Germany.","DOI":"10.1109\/ICRA.2013.6631094"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Schneider, N., Piewak, F., Stiller, C., and Franke, U. (2017, January 11\u201314). RegNet: Multimodal sensor registration using deep neural networks. Proceedings of the 2017 IEEE Intelligent Vehicles Symposium (IV), Los Angeles, CA, USA.","DOI":"10.1109\/IVS.2017.7995968"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Moulon, P., Monasse, P., and Marlet, R. (2013, January 1\u20138). Global Fusion of Relative Motions for Robust, Accurate and Scalable Structure from Motion. Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia.","DOI":"10.1109\/ICCV.2013.403"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Nagy, B., and Benedek, C. (2018, January 14). Real-time point cloud alignment for vehicle localization in a high resolution 3D map. Proceedings of the Workshop on Computer Vision for Road Scene Understanding and Autonomous Driving (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-11009-3_13"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"He, K., Gkioxari, G., Doll\u00e1r, P., and Girshick, R. (2017, January 22\u201329). Mask R-CNN. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.322"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Lang, A.H., Vora, S., Caesar, H., Zhou, L., Yang, J., and Beijbom, O. (2019, January 16\u201320). PointPillars: Fast Encoders for Object Detection From Point Clouds. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.01298"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1007\/s11263-008-0152-6","article-title":"EPnP: An Accurate O(n) Solution to the PnP Problem","volume":"81","author":"Lepetit","year":"2008","journal-title":"Int. J. Comput. Vis."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Moulon, P., Monasse, P., Perrot, R., and Marlet, R. (2016, January 4). OpenMVG: Open Multiple View Geometry. Proceedings of the Workshop on Reproducible Research in Pattern Recognition, Canc\u00fan, Mexico.","DOI":"10.1007\/978-3-319-56414-2_5"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1145\/1531326.1531330","article-title":"PatchMatch: A randomized correspondence algorithm for structural image editing","volume":"28","author":"Barnes","year":"2009","journal-title":"ACM Trans. Graph."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Magnusson, M., Nuchter, A., Lorken, C., Lilienthal, A.J., and Hertzberg, J. (2009, January 12\u201317). Evaluation of 3D registration reliability and speed\u2014A comparison of ICP and NDT. Proceedings of the IEEE International Conference on Robotics and Automation, Kobe, Japan.","DOI":"10.1109\/ROBOT.2009.5152538"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Rusu, R., and Cousins, S. (2011, January 9\u201313). 3D is here: Point cloud library (PCL). Proceedings of the International Conference on Robotics and Automation, Shanghai, China.","DOI":"10.1109\/ICRA.2011.5980567"},{"key":"ref_30","first-page":"628","article-title":"Fast 3-D Urban Object Detection on Streaming Point Clouds","volume":"Volume 8926","author":"Nagy","year":"2015","journal-title":"Workshop on Computer Vision for Road Scene Understanding and Autonomous Driving at ECCV\u201914"},{"key":"ref_31","first-page":"1048","article-title":"Efficient Sparse Voxel Octrees","volume":"17","author":"Samuli","year":"2010","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"ref_32","first-page":"101:1","article-title":"High Resolution Sparse Voxel DAGs","volume":"32","author":"Viktor","year":"2013","journal-title":"ACM Trans. Graph."},{"key":"ref_33","unstructured":"Matti, L., Anttoni, J., Juha, H., Jouko, L., Harri, K., Antero, K., Eetu, P., and Hannu, H. (2015). Object Classification and Recognition From Mobile Laser Scanning Point Clouds in a Road Environment. IEEE Trans. Geosci. Remote Sens."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"164","DOI":"10.1090\/qam\/10666","article-title":"A method for the solution of certain non-linear problems in least squares","volume":"2","author":"Levenberg","year":"1944","journal-title":"Q. Appl. Math."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Kim, E.S., and Park, S.Y. (2019). Extrinsic Calibration between Camera and LiDAR Sensors by Matching Multiple 3D Planes. Sensors, 20.","DOI":"10.3390\/s20010052"},{"key":"ref_36","unstructured":"B\u00f6rcs, A., Nagy, B., Baticz, M., and Benedek, C. (2014). A Model-based Approach for Fast Vehicle Detection in Continuously Streamed Urban LIDAR Point Clouds. Workshop on Scene Understanding for Autonomous Systems at ACCV, Lecture Notes in Computer Science, Springer Science+Business Media."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/12\/7\/1137\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:14:55Z","timestamp":1760174095000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/12\/7\/1137"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,4,2]]},"references-count":36,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2020,4]]}},"alternative-id":["rs12071137"],"URL":"https:\/\/doi.org\/10.3390\/rs12071137","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,4,2]]}}}