{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,15]],"date-time":"2026-01-15T22:12:27Z","timestamp":1768515147867,"version":"3.49.0"},"reference-count":39,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2023,6,17]],"date-time":"2023-06-17T00:00:00Z","timestamp":1686960000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100004329","name":"Slovenian Research Agency","doi-asserted-by":"publisher","award":["J2-2506"],"award-info":[{"award-number":["J2-2506"]}],"id":[{"id":"10.13039\/501100004329","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004329","name":"Slovenian Research Agency","doi-asserted-by":"publisher","award":["P2-0095"],"award-info":[{"award-number":["P2-0095"]}],"id":[{"id":"10.13039\/501100004329","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Multimodal sensor systems require precise calibration if they are to be used in the field. Due to the difficulty of obtaining the corresponding features from different modalities, the calibration of such systems is an open problem. We present a systematic approach for calibrating a set of cameras with different modalities (RGB, thermal, polarization, and dual-spectrum near infrared) with regard to a LiDAR sensor using a planar calibration target. Firstly, a method for calibrating a single camera with regard to the LiDAR sensor is proposed. The method is usable with any modality, as long as the calibration pattern is detected. A methodology for establishing a parallax-aware pixel mapping between different camera modalities is then presented. Such a mapping can then be used to transfer annotations, features, and results between highly differing camera modalities to facilitate feature extraction and deep detection and segmentation methods.<\/jats:p>","DOI":"10.3390\/s23125676","type":"journal-article","created":{"date-parts":[[2023,6,19]],"date-time":"2023-06-19T02:29:19Z","timestamp":1687141759000},"page":"5676","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Joint Calibration of a Multimodal Sensor System for Autonomous Vehicles"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4082-2506","authenticated-orcid":false,"given":"Jon","family":"Muhovi\u010d","sequence":"first","affiliation":[{"name":"Faculty of Electrical Engineering, University of Ljubljana, Tr\u017ea\u0161ka Cesta 25, SI-1000 Ljubljana, Slovenia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6039-6110","authenticated-orcid":false,"given":"Janez","family":"Per\u0161","sequence":"additional","affiliation":[{"name":"Faculty of Electrical Engineering, University of Ljubljana, Tr\u017ea\u0161ka Cesta 25, SI-1000 Ljubljana, Slovenia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,6,17]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1231","DOI":"10.1177\/0278364913491297","article-title":"Vision meets robotics: The kitti dataset","volume":"32","author":"Geiger","year":"2013","journal-title":"Int. J. Robot. Res."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1177\/0278364916679498","article-title":"1 year, 1000 km: The Oxford RobotCar dataset","volume":"36","author":"Maddern","year":"2017","journal-title":"Int. J. Robot. Res."},{"key":"ref_3","unstructured":"Qi, C.R., Su, H., Mo, K., and Guibas, L.J. (2017, January 21\u201326). Pointnet: Deep learning on point sets for 3d classification and segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Zhao, H., Jiang, L., Jia, J., Torr, P.H., and Koltun, V. (2021, January 20\u201325). Point transformer. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Nashville, TN, USA.","DOI":"10.1109\/ICCV48922.2021.01595"},{"key":"ref_5","unstructured":"Zhang, Q., and Pless, R. (October, January 28). Extrinsic calibration of a camera and laser range finder (improves camera calibration). Proceedings of the 2004 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No. 04CH37566), Sendai, Japan."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"336","DOI":"10.3182\/20100906-3-IT-2019.00059","article-title":"Extrinsic calibration of a 3d laser scanner and an omnidirectional camera","volume":"43","author":"Pandey","year":"2010","journal-title":"IFAC Proc. Volume"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Guindel, C., Beltr\u00e1n, J., Mart\u00edn, D., and Garc\u00eda, F. (2017, January 16\u201319). Automatic extrinsic calibration for lidar-stereo vehicle sensor setups. Proceedings of the 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), Yokohama, Japan.","DOI":"10.1109\/ITSC.2017.8317829"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"17677","DOI":"10.1109\/TITS.2022.3155228","article-title":"Automatic extrinsic calibration method for lidar and camera sensor setups","volume":"23","author":"Guindel","year":"2022","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Pusztai, Z., Eichhardt, I., and Hajder, L. (2018). Accurate calibration of multi-lidar-multi-camera systems. Sensors, 18.","DOI":"10.3390\/s18072139"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Grammatikopoulos, L., Papanagnou, A., Venianakis, A., Kalisperakis, I., and Stentoumis, C. (2022). An Effective Camera-to-Lidar Spatiotemporal Calibration Based on a Simple Calibration Target. Sensors, 22.","DOI":"10.3390\/s22155576"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Ou, J., Huang, P., Zhou, J., Zhao, Y., and Lin, L. (2022). Automatic Extrinsic Calibration of 3D LIDAR and Multi-Cameras Based on Graph Optimization. Sensors, 22.","DOI":"10.3390\/s22062221"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"222","DOI":"10.1016\/j.icte.2021.12.016","article-title":"A sensor fusion system with thermal infrared camera and LiDAR for autonomous vehicles and deep learning based object detection","volume":"9","author":"Choi","year":"2023","journal-title":"ICT Express"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1007\/s11263-008-0152-6","article-title":"EP n P: An accurate O (n) solution to the P n P problem","volume":"81","author":"Lepetit","year":"2009","journal-title":"Int. J. Comput. Vis."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Levinson, J., and Thrun, S. (2013, January 24\u201328). Automatic Online Calibration of Cameras and Lasers. Proceedings of the Robotics: Science and Systems, Berlin, Germany.","DOI":"10.15607\/RSS.2013.IX.029"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Pandey, G., McBride, J.R., Savarese, S., and Eustice, R.M. (2012, January 22\u201326). Automatic targetless extrinsic calibration of a 3d lidar and camera by maximizing mutual information. Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence, Toronto, ON, Canada.","DOI":"10.1609\/aaai.v26i1.8379"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Iyer, G., Ram, R.K., Murthy, J.K., and Krishna, K.M. (2018, January 1\u20135). CalibNet: Geometrically supervised extrinsic calibration using 3D spatial transformer networks. Proceedings of the 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.","DOI":"10.1109\/IROS.2018.8593693"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Lv, X., Wang, B., Dou, Z., Ye, D., and Wang, S. (2021, January 20\u201325). LCCNet: LiDAR and camera self-calibration using cost volume network. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPRW53098.2021.00324"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Lv, X., Wang, S., and Ye, D. (2021). CFNet: LiDAR-camera registration using calibration flow network. Sensors, 21.","DOI":"10.3390\/s21238112"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"7517","DOI":"10.1109\/LRA.2021.3098923","article-title":"Pixel-level extrinsic self calibration of high resolution lidar and camera in targetless environments","volume":"6","author":"Yuan","year":"2021","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Zhu, Y., Li, C., and Zhang, Y. (August, January 31). Online camera-lidar calibration with sensor semantic information. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France.","DOI":"10.1109\/ICRA40945.2020.9196627"},{"key":"ref_21","unstructured":"Wang, W., Nobuhara, S., Nakamura, R., and Sakurada, K. (2020). Soic: Semantic online initialization and calibration for lidar and camera. arXiv."},{"key":"ref_22","unstructured":"Takahashi, H., and Tomita, F. (1988, January 5\u20138). Self-calibration of stereo cameras. Proceedings of the 1988 Second International Conference on Computer Vision, Tampa, FL, USA."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Hwang, S., Park, J., Kim, N., Choi, Y., and So Kweon, I. (2015, January 7\u201312). Multispectral pedestrian detection: Benchmark dataset and baseline. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298706"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Rangel, J., Soldan, S., and Kroll, A. (2014, January 7\u201311). 3D thermal imaging: Fusion of thermography and depth cameras. Proceedings of the International Conference on Quantitative InfraRed Thermography, Bordeaux, France.","DOI":"10.21611\/qirt.2014.035"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Shivakumar, S.S., Rodrigues, N., Zhou, A., Miller, I.D., Kumar, V., and Taylor, C.J. (August, January 31). Pst900: Rgb-thermal calibration, dataset and segmentation network. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France.","DOI":"10.1109\/ICRA40945.2020.9196831"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B. (2016, January 27\u201330). The cityscapes dataset for semantic urban scene understanding. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.350"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. (2020, January 13\u201319). nuscenes: A multimodal dataset for autonomous driving. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01164"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., and Caine, B. (2020, January 13\u201319). Scalability in perception for autonomous driving: Waymo open dataset. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00252"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Jia, X., Zhu, C., Li, M., Tang, W., and Zhou, W. (2021, January 20\u201325). LLVIP: A Visible-infrared Paired Dataset for Low-light Vision. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Nashville, TN, USA.","DOI":"10.1109\/ICCVW54120.2021.00389"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Gonz\u00e1lez, A., Fang, Z., Socarras, Y., Serrat, J., V\u00e1zquez, D., Xu, J., and L\u00f3pez, A.M. (2016). Pedestrian detection at day\/night time with visible and FIR cameras: A comparison. Sensors, 16.","DOI":"10.3390\/s16060820"},{"key":"ref_31","unstructured":"Lee, A.J., Cho, Y., Yoon, S., Shin, Y., and Kim, A. (2019, January 20\u201324). ViViD: Vision for visibility dataset. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) Workshop: Dataset Generation and Benchmarking of SLAM Algorithms for Robotics and VR\/AR, Montreal, QC, Canada."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"6282","DOI":"10.1109\/LRA.2022.3168335","article-title":"ViViD++: Vision for Visibility Dataset","volume":"7","author":"Lee","year":"2022","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_33","unstructured":"Kniaz, V.V., Knyaz, V.A., Hladuvka, J., Kropatsch, W.G., and Mizginov, V. (, January 8\u201314). Thermalgan: Multimodal color-to-thermal image translation for person re-identification in multispectral dataset. Proceedings of the European Conference on Computer Vision (ECCV) Workshops, Munich, Germany."},{"key":"ref_34","unstructured":"Per\u0161, J., Muhovi\u010d, J., Bobek, U., Cvenkel, T., Gregorin, D., Mitja, K., Lukek, M., Sedej, N., and Kristan, M. (2021, January 20\u201321). Modular Multi-Sensor System for Unmanned Surface Vehicles. Proceedings of the 30th International Electrotechnical and Computer Science Conference ERK2021, Portoro\u017e, Slovenia."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1330","DOI":"10.1109\/34.888718","article-title":"A flexible new technique for camera calibration","volume":"22","author":"Zhang","year":"2000","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_36","unstructured":"Wright, G.B. (2003). Radial Basis Function Interpolation: Numerical and Analytical Developments, University of Colorado at Boulder."},{"key":"ref_37","unstructured":"Wang, C.Y., Bochkovskiy, A., and Liao, H.Y.M. (2022). YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. arXiv."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"12661","DOI":"10.1109\/TCYB.2021.3085856","article-title":"WaSR\u2014A water segmentation and refinement maritime obstacle detection network","volume":"52","author":"Bovcon","year":"2021","journal-title":"IEEE Trans. Cybern."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1007\/s11263-009-0275-4","article-title":"The pascal visual object classes (voc) challenge","volume":"88","author":"Everingham","year":"2010","journal-title":"Int. J. Comput. Vis."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/12\/5676\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:55:35Z","timestamp":1760126135000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/12\/5676"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,17]]},"references-count":39,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2023,6]]}},"alternative-id":["s23125676"],"URL":"https:\/\/doi.org\/10.3390\/s23125676","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,17]]}}}