{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,5]],"date-time":"2026-03-05T15:30:19Z","timestamp":1772724619290,"version":"3.50.1"},"reference-count":28,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2022,5,25]],"date-time":"2022-05-25T00:00:00Z","timestamp":1653436800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Aptiv Technical Center Krakow"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>In this article, we propose a method for automatic calibration of a LiDAR\u2013camera system, which can be used in autonomous cars. This approach does not require any calibration pattern, as calibration is only based on real traffic scenes observed by sensors; the results of camera image segmentation are compared with scanning LiDAR depth data. The proposed algorithm superimposes the edges of objects segmented by the Mask-RCNN network with depth discontinuities. The method can run in the background during driving, and it can automatically detect decalibration and correct corresponding rotation matrices in an online and near real-time mode. Experiments on the KITTI dataset demonstrated that, for input data of moderate quality, the algorithm could calculate and correct rotation matrices with an average accuracy of 0.23\u00b0.<\/jats:p>","DOI":"10.3390\/rs14112531","type":"journal-article","created":{"date-parts":[[2022,5,31]],"date-time":"2022-05-31T00:25:12Z","timestamp":1653956712000},"page":"2531","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["Automatic Calibration of a LiDAR\u2013Camera System Based on Instance Segmentation"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1556-6539","authenticated-orcid":false,"given":"Pawel","family":"Rotter","sequence":"first","affiliation":[{"name":"Department of Automatic Control and Robotics, AGH University of Science and Technology, Al. A. Mickiewicza 30, 30-059 Krakow, Poland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0052-7083","authenticated-orcid":false,"given":"Maciej","family":"Klemiato","sequence":"additional","affiliation":[{"name":"Department of Automatic Control and Robotics, AGH University of Science and Technology, Al. A. Mickiewicza 30, 30-059 Krakow, Poland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8290-8375","authenticated-orcid":false,"given":"Pawel","family":"Skruch","sequence":"additional","affiliation":[{"name":"Department of Automatic Control and Robotics, AGH University of Science and Technology, Al. A. Mickiewicza 30, 30-059 Krakow, Poland"}]}],"member":"1968","published-online":{"date-parts":[[2022,5,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Geiger, A., Moosmann, F., Car, \u00d6., and Schuster, B. (2012, January 14\u201318). Automatic camera and range sensor calibration using a single shot. Proceedings of the 2012 IEEE International Conference on Robotics and Automation, Saint Paul, MN, USA.","DOI":"10.1109\/ICRA.2012.6224570"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Zhou, L., Li, Z., and Kaess, M. (2018, January 1\u20135). Automatic Extrinsic Calibration of a Camera and a 3D LiDAR Using Line and Plane Correspondences. Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.","DOI":"10.1109\/IROS.2018.8593660"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Alismail, H., Baker, L.D., and Browning, B. (2012, January 13\u201315). Automatic Calibration of a Range Sensor and Camera System. Proceedings of the 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission, Zurich, Switzerland.","DOI":"10.1109\/3DIMPVT.2012.52"},{"key":"ref_4","unstructured":"Martin, V., \u0160pan\u011bl, M., Materna, Z., and Herout, A. (2014, January 2\u20136). Calibration of RGB Camera with Velodyne LiDAR. Proceedings of the 21st International Conference on Computer Graphics, Visualization and Computer Vision, Plzen, Czech Republic."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"5333","DOI":"10.3390\/s140305333","article-title":"Calibration between Color Camera and 3D LIDAR Instruments with a Polygonal Planar Board","volume":"14","author":"Park","year":"2014","journal-title":"Sensors"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1902","DOI":"10.3390\/s130201902","article-title":"3D LIDAR-Camera Extrinsic Calibration Using an Arbitrary Trihedron","volume":"13","author":"Gong","year":"2013","journal-title":"Sensors"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Pusztai, Z., and Hajder, L. (2017, January 22\u201329). Accurate calibration of lidar-camera systems using ordinary boxes. Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW), Venice, Italy.","DOI":"10.1109\/ICCVW.2017.53"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Pusztai, Z., Eichhardt, I., and Hajder, L. (2018). Accurate calibration of multi-lidar-multi-camera systems. Sensors, 18.","DOI":"10.3390\/s18072139"},{"key":"ref_9","unstructured":"T\u00f3th, T., Pusztai, Z., and Hajder, L. (2010, January 23\u201327). Automatic LiDAR-Camera Calibration of Extrinsic Parameters Using a Spherical Target. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Paris, France."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"696","DOI":"10.1002\/rob.21542","article-title":"Automatic Extrinsic Calibration of Vision and Lidar by Maximizing Mutual Information","volume":"32","author":"Pandey","year":"2014","journal-title":"J. Field Robot."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Ishikawa, R., Oishi, T., and Ikeuchi, K. (2018, January 1\u20135). Lidar and camera calibration using motions estimated by sensor fusion odometry. Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.","DOI":"10.1109\/IROS.2018.8593360"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Minaee, S., Boykov, Y.Y., Porikli, F., Plaza, A.J., Kehtarnavaz, N., and Terzopoulos, D. (2020). Image Segmentation Using Deep Learning: A Survey. arXiv.","DOI":"10.1109\/TPAMI.2021.3059968"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"He, K., Gkioxari, G., Doll\u00e1r, P., and Girshick, R. (2017, January 22\u201329). Mask R-CNN. Proceedings of the International Conference of Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.322"},{"key":"ref_14","unstructured":"Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., and Girshick, R. (2022, March 17). Detectron2. Available online: https:\/\/github.com\/facebookresearch\/detectron2."},{"key":"ref_15","unstructured":"Ren, S., He, K., Girshick, R., and Sun, J. (2015, January 7\u201312). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Proceedings of the 28th International Conference on Neural Information Processing Systems 28 (NIPS 2015), Montreal, QC, Canada."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Haris, M., and Glowacz, A. (2021). Road Object Detection: A Comparative Study of Deep Learning-Based Algorithms. Electronics, 10.","DOI":"10.3390\/electronics10161932"},{"key":"ref_17","unstructured":"Ma, W.Y., and Manjunath, B.S. (1997, January 17\u201319). Edge Flow\u2014A framework of boundary detection and image segmentation. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, San Juan, Puerto Rico."},{"key":"ref_18","unstructured":"Panda, S., and Nanda, P.K. (2018). Color Image Segmentation Using Markov Random Field Models, Lambert Academic Publishing."},{"key":"ref_19","unstructured":"Masoud, M., and Sood, R. (2018). Instance segmentation using depth and mask RCNN. CS230: Deep Learning Project Reports and Posters, Stanford University. Available online: https:\/\/cs230.stanford.edu\/projects_spring_2018\/reports\/8285407.pdf."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Danielczuk, M., Matl, M., Gupta, S., Li, A., Lee, A., Mahler, J., and Goldberg, K. (2019, January 20\u201324). Segmenting Unknown 3D Objects from Real Depth Images using Mask R-CNN Trained on Synthetic Data. Proceedings of the International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.","DOI":"10.1109\/ICRA.2019.8793744"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Zhou, Y., and Tuzel, O. (2018, January 18\u201323). VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00472"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Ali, W., Abdelkarim, S., Zidan, M., Zahran, M., and El Sallab, A. (2018, January 8\u201314). YOLO3D: End-to-end real-time 3D Oriented Object Bounding Box Detection from LiDAR Point Cloud. Proceedings of the ECCV 2018: Computer Vision\u2013ECCV 2018 Workshops, Munich, Germany.","DOI":"10.1007\/978-3-030-11015-4_54"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Lang, A.H., Vora, S., Caesar, H., Zhou, L., Yang, J., and Beijbom, O. (2019). PointPillars: Fast Encoders for Object Detection from Point Clouds. arXiv.","DOI":"10.1109\/CVPR.2019.01298"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Dinh, T.H., Pham, M.T., Phung, M.D., Nguyen, D.M., Hoang, V.M., and Tran, Q.V. (2014, January 10\u201312). Image segmentation based on histogram of depth and an application in driver distraction detection. Proceedings of the 13th International Conference on Control Automation Robotics & Vision (ICARCV), Singapore.","DOI":"10.1109\/ICARCV.2014.7064437"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1231","DOI":"10.1177\/0278364913491297","article-title":"Vision meets robotics: The KITTI dataset","volume":"32","author":"Geiger","year":"2013","journal-title":"Int. J. Robot. Res."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. (2020, January 13\u201319). nuScenes: A multimodal dataset for autonomous driving. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01164"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"889","DOI":"10.1137\/S1052623400378742","article-title":"Analysis of Generalized Pattern Searches","volume":"13","author":"Audet","year":"2002","journal-title":"SIAM J. Optim."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Wu, X., Wen, S., and Xie, Y.-A. (2019). Improvement of Mask-RCNN Object Segmentation Algorithm. Intelligent Robotics and Applications, Springer.","DOI":"10.1007\/978-3-030-27526-6_51"}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/14\/11\/2531\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:18:43Z","timestamp":1760138323000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/14\/11\/2531"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,25]]},"references-count":28,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2022,6]]}},"alternative-id":["rs14112531"],"URL":"https:\/\/doi.org\/10.3390\/rs14112531","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,5,25]]}}}