{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:37:17Z","timestamp":1760143037176,"version":"build-2065373602"},"reference-count":36,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2024,1,20]],"date-time":"2024-01-20T00:00:00Z","timestamp":1705708800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant"},{"name":"Computer Science Department at the University of Western Ontario, Canada"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>LiDAR sensors, pivotal in various fields like agriculture and robotics for tasks such as 3D object detection and map creation, are increasingly coupled with thermal cameras to harness heat information. This combination proves particularly effective in adverse conditions like darkness and rain. Ensuring seamless fusion between the sensors necessitates precise extrinsic calibration. Our innovative calibration method leverages human presence during sensor setup movements, eliminating the reliance on dedicated calibration targets. It optimizes extrinsic parameters by employing a novel evolutionary algorithm on a specifically designed loss function that measures human alignment across modalities. Our approach showcases a notable 4.43% improvement in the loss over extrinsic parameters obtained from target-based calibration in the FieldSAFE dataset. This advancement reduces costs related to target creation, saves time in diverse pose collection, mitigates repetitive calibration efforts amid sensor drift or setting changes, and broadens accessibility by obviating the need for specific targets. The adaptability of our method in various environments, like urban streets or expansive farm fields, stems from leveraging the ubiquitous presence of humans. Our method presents an efficient, cost-effective, and readily applicable means of extrinsic calibration, enhancing sensor fusion capabilities in the critical fields reliant on precise and robust data acquisition.<\/jats:p>","DOI":"10.3390\/s24020669","type":"journal-article","created":{"date-parts":[[2024,1,22]],"date-time":"2024-01-22T11:36:41Z","timestamp":1705923401000},"page":"669","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Extrinsic Calibration of Thermal Camera and 3D LiDAR Sensor via Human Matching in Both Modalities during Sensor Setup Movement"],"prefix":"10.3390","volume":"24","author":[{"given":"Farhad","family":"Dalirani","sequence":"first","affiliation":[{"name":"Computer Science Department, Western University, London, ON N6A 3K7, Canada"}]},{"given":"Mahmoud R.","family":"El-Sakka","sequence":"additional","affiliation":[{"name":"Computer Science Department, Western University, London, ON N6A 3K7, Canada"}]}],"member":"1968","published-online":{"date-parts":[[2024,1,20]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Koci\u0107, J., Jovi\u010di\u0107, N., and Drndarevi\u0107, V. (2018, January 20\u201321). Sensors and sensor fusion in autonomous vehicles. Proceedings of the 2018 26th Telecommunications Forum (TELFOR), Belgrade, Serbia.","DOI":"10.1109\/TELFOR.2018.8612054"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1029","DOI":"10.1109\/LRA.2023.3236571","article-title":"Kiss-icp: In defense of point-to-point icp\u2013simple, accurate, and robust registration if done the right way","volume":"8","author":"Vizzo","year":"2023","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_3","unstructured":"Behley, J., Garbade, M., Milioto, A., Quenzel, J., Behnke, S., Stachniss, C., and Gall, J. (November, January 27). Semantickitti: A dataset for semantic scene understanding of lidar sequences. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Seoul, Republic of Korea."},{"key":"ref_4","unstructured":"Lang, A.H., Vora, S., Caesar, H., Zhou, L., Yang, J., and Beijbom, O. (November, January 27). Pointpillars: Fast encoders for object detection from point clouds. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seoul, Republic of Korea."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"232","DOI":"10.1109\/MGRS.2020.3032713","article-title":"Lidar boosts 3D ecological observations and modelings: A review and perspective","volume":"9","author":"Guo","year":"2020","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Debnath, S., Paul, M., and Debnath, T. (2023). Applications of LiDAR in Agriculture and Future Research Directions. J. Imaging, 9.","DOI":"10.3390\/jimaging9030057"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Xu, X., Dong, S., Xu, T., Ding, L., Wang, J., Jiang, P., Song, L., and Li, J. (2023). FusionRCNN: LiDAR-Camera Fusion for Two-Stage 3D Object Detection. Remote Sens., 15.","DOI":"10.3390\/rs15071839"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Miethig, B., Liu, A., Habibi, S., and Mohrenschildt, M.V. (2019, January 19\u201321). Leveraging thermal imaging for autonomous driving. Proceedings of the 2019 IEEE Transportation Electrification Conference and Expo (ITEC), Detroit, MI, USA.","DOI":"10.1109\/ITEC.2019.8790493"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"186","DOI":"10.1007\/s11947-010-0333-5","article-title":"Applications of thermal imaging in agriculture and food industry\u2014A review","volume":"4","author":"Vadivambal","year":"2011","journal-title":"Food Bioprocess Technol."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1007\/s00138-013-0570-5","article-title":"Thermal cameras and applications: A survey","volume":"25","author":"Gade","year":"2014","journal-title":"Mach. Vis. Appl."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Kragh, M.F., Christiansen, P., Laursen, M.S., Larsen, M., Steen, K.A., Green, O., Karstoft, H., and J\u00f8rgensen, R.N. (2017). Fieldsafe: Dataset for obstacle detection in agriculture. Sensors, 17.","DOI":"10.3390\/s17112579"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"934","DOI":"10.1109\/TITS.2018.2791533","article-title":"KAIST multi-spectral day\/night data set for autonomous and assisted driving","volume":"19","author":"Choi","year":"2018","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Shin, U., Park, J., and Kweon, I.S. (2023, January 18\u201322). Deep Depth Estimation From Thermal Image. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.","DOI":"10.1109\/CVPR52729.2023.00107"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"2266","DOI":"10.1109\/LRA.2021.3138527","article-title":"M2dgr: A multi-sensor and multi-scenario slam dataset for ground robots","volume":"7","author":"Yin","year":"2021","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"101712","DOI":"10.1016\/j.mex.2022.101712","article-title":"An approach for monitoring temperature on fruit surface by means of thermal point cloud","volume":"9","author":"Tsoulias","year":"2022","journal-title":"MethodsX"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Yue, Y., Yang, C., Zhang, J., Wen, M., Wu, Z., Zhang, H., and Wang, D. (August, January 31). Day and night collaborative dynamic mapping in unstructured environment based on multimodal sensors. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France.","DOI":"10.1109\/ICRA40945.2020.9197072"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Geiger, A., Moosmann, F., Car, \u00d6., and Schuster, B. (2012, January 14\u201318). Automatic camera and range sensor calibration using a single shot. Proceedings of the 2012 IEEE International Conference on Robotics and Automation, Saint Paul, MN, USA.","DOI":"10.1109\/ICRA.2012.6224570"},{"key":"ref_18","unstructured":"Krishnan, A.K., Stinnett, B., and Saripalli, S. (October, January 28). Cross-calibration of rgb and thermal cameras with a lidar. Proceedings of the IROS 2015 Workshop on Alternative Sensing for Robot Perception, Hamburg, Germany."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1142\/S2301385017500054","article-title":"Cross-calibration of rgb and thermal cameras with a lidar for rgb-depth-thermal mapping","volume":"5","author":"Krishnan","year":"2017","journal-title":"Unmanned Syst."},{"key":"ref_20","unstructured":"Borrmann, D. (2018). Multi-Modal 3D Mapping-Combining 3D Point Clouds with Thermal and Color Information, Universit\u00e4t W\u00fcrzburg."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Dalirani, F., Heidari, F., Rahman, T., Cheema, D.S., and Bauer, M.A. (2023, January 4\u20137). Automatic Extrinsic Calibration of Thermal Camera and LiDAR for Vehicle Sensor Setups. Proceedings of the 2023 IEEE Intelligent Vehicles Symposium (IV), Anchorage, AK, USA.","DOI":"10.1109\/IV55152.2023.10186694"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Zhang, J., Liu, Y., Wen, M., Yue, Y., Zhang, H., and Wang, D. (2023, January 4\u20137). L2V2T2Calib: Automatic and Unified Extrinsic Calibration Toolbox for Different 3D LiDAR, Visual Camera and Thermal Camera. Proceedings of the 2023 IEEE Intelligent Vehicles Symposium (IV), Anchorage, AK, USA.","DOI":"10.1109\/IV55152.2023.10186657"},{"key":"ref_23","unstructured":"Azam, S., Munir, F., Sheri, A.M., Ko, Y., Hussain, I., and Jeon, M. (2019). Applied Industrial Optics: Spectroscopy, Imaging and Metrology, Optica Publishing Group."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Zhang, J., Siritanawan, P., Yue, Y., Yang, C., Wen, M., and Wang, D. (2018, January 18\u201321). A two-step method for extrinsic calibration between a sparse 3d lidar and a thermal camera. Proceedings of the 2018 15th International Conference on Control, Automation, Robotics and Vision (ICARCV), Singapore.","DOI":"10.1109\/ICARCV.2018.8581170"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"350","DOI":"10.1007\/s11119-017-9497-6","article-title":"Platform for evaluating sensors and human detection in autonomous mowing operations","volume":"18","author":"Christiansen","year":"2017","journal-title":"Precis. Agric."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1007\/BF01427149","article-title":"Iterative point matching for registration of free-form curves and surfaces","volume":"13","author":"Zhang","year":"1994","journal-title":"Int. J. Comput. Vis."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Shin, U., Park, J., and Kweon, I.S. (2023, December 10). Supplementary Material: Deep Depth Estimation from Thermal Image. Available online: https:\/\/openaccess.thecvf.com\/content\/CVPR2023\/supplemental\/Shin_Deep_Depth_Estimation_CVPR_2023_supplemental.pdf.","DOI":"10.1109\/CVPR52729.2023.00107"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Fu, T., Yu, H., Yang, W., Hu, Y., and Scherer, S. (2021). Targetless Extrinsic Calibration of Stereo Cameras, Thermal Cameras, and Laser Sensors in the Wild. arXiv.","DOI":"10.1109\/TIM.2022.3204338"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"679","DOI":"10.1109\/TPAMI.1986.4767851","article-title":"A computational approach to edge detection","volume":"PAMI-8","author":"Canny","year":"1986","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Mharolkar, S., Zhang, J., Peng, G., Liu, Y., and Wang, D. (2022, January 8\u201312). RGBDTCalibNet: End-to-end Online Extrinsic Calibration between a 3D LiDAR, an RGB Camera and a Thermal Camera. Proceedings of the 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), Macau, China.","DOI":"10.1109\/ITSC55140.2022.9922437"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Xie, J., Kiefel, M., Sun, M.T., and Geiger, A. (2016, January 27\u201330). Semantic instance annotation of street scenes by 3d to 2d label transfer. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.401"},{"key":"ref_32","first-page":"120","article-title":"The openCV library","volume":"25","author":"Bradski","year":"2000","journal-title":"Dr. Dobb\u2019S J. Softw. Tools Prof. Program."},{"key":"ref_33","unstructured":"Ren, S., He, K., Girshick, R., and Sun, J. (2015, January 7\u201312). Faster r-cnn: Towards real-time object detection with region proposal networks. Proceedings of the Advances in Neural Information Processing Systems (NIPS 2015), Montreal, QC, Canada."},{"key":"ref_34","unstructured":"Teledyne, F. (2018). Free Teledyne FLIR Thermal Dataset for Algorithm Training, Teledyne FLIR."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., and Lo, W.Y. (2023). Segment anything. arXiv.","DOI":"10.1109\/ICCV51070.2023.00371"},{"key":"ref_36","unstructured":"Contributors, M. (2023, December 10). OpenMMLab\u2019s Next-Generation Platform for General 3D Object Detection. Available online: https:\/\/github.com\/open-mmlab\/mmdetection3d."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/2\/669\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T13:46:32Z","timestamp":1760103992000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/2\/669"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,20]]},"references-count":36,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2024,1]]}},"alternative-id":["s24020669"],"URL":"https:\/\/doi.org\/10.3390\/s24020669","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2024,1,20]]}}}