{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,30]],"date-time":"2026-07-30T11:21:54Z","timestamp":1785410514070,"version":"3.56.0"},"reference-count":44,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2026,5,22]],"date-time":"2026-05-22T00:00:00Z","timestamp":1779408000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["No. 62127813"],"award-info":[{"award-number":["No. 62127813"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"award":["No. 62127813"],"award-info":[{"award-number":["No. 62127813"]}],"id":[{"id":"https:\/\/ror.org\/01h0zpd94","id-type":"ROR","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"Joint Funds of the National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["No. U2341226"],"award-info":[{"award-number":["No. U2341226"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Jilin Provincial Talent Special Project","award":["No. 20240602015RC"],"award-info":[{"award-number":["No. 20240602015RC"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Imaging"],"abstract":"<jats:p>To address the limitations of traditional methods in feature extraction and multi-modal information fusion, this paper proposes an infrared\u2013visible image object detection architecture that integrates Convolutional Neural Networks (CNNs) and Deformable Transformers. This method leverages the advantages of CNN in local feature modeling and the capabilities of Transformer in capturing global contextual information, facilitating the fusion of semantic consistency and structural details across modalities. By introducing a detection-aware multi-task optimization mechanism, the model improves object detection in challenging scenarios such as low-light conditions, occlusion, and complex backgrounds. Experiments on multiple standard datasets, including M3FD and LLVIP, indicate that the proposed method achieves competitive or better performance than the compared methods in key metrics such as mAP. Specifically, our method obtains the best mAP50 among the evaluated methods with an mAP50 of 74.2% on the M3FD dataset and 98.6% on the LLVIP dataset, surpassing the second-best PIAFusion by 4.3% and 2.5% respectively. These quantitative results support the practicality and effectiveness of our approach in the evaluated complex environments.<\/jats:p>","DOI":"10.3390\/jimaging12060219","type":"journal-article","created":{"date-parts":[[2026,5,22]],"date-time":"2026-05-22T12:14:52Z","timestamp":1779452092000},"page":"219","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Visible\u2013Infrared Fusion Based on CNN and Deformable Transformer"],"prefix":"10.3390","volume":"12","author":[{"given":"Xiaoyi","family":"Wang","sequence":"first","affiliation":[{"name":"College of Opto-Electronic Engineering, Changchun University of Science and Technology, Changchun 130022, China"},{"name":"State Key Laboratory of Dynamic Optical Imaging and Measurement, Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun 130033, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiansong","family":"Gu","sequence":"additional","affiliation":[{"name":"College of Opto-Electronic Engineering, Changchun University of Science and Technology, Changchun 130022, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bin","family":"Li","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Dynamic Optical Imaging and Measurement, Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun 130033, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mingqiang","family":"Zhang","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Dynamic Optical Imaging and Measurement, Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun 130033, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Panpan","family":"Yang","sequence":"additional","affiliation":[{"name":"College of Opto-Electronic Engineering, Changchun University of Science and Technology, Changchun 130022, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qiang","family":"Fu","sequence":"additional","affiliation":[{"name":"College of Opto-Electronic Engineering, Changchun University of Science and Technology, Changchun 130022, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,5,22]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1016\/j.inffus.2018.02.004","article-title":"Infrared and visible image fusion methods and applications: A survey","volume":"45","author":"Ma","year":"2019","journal-title":"Inf. Fusion"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"2614","DOI":"10.1109\/TIP.2018.2887342","article-title":"DenseFuse: A fusion approach to infrared and visible images","volume":"28","author":"Li","year":"2018","journal-title":"IEEE Trans. Image Process."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1016\/j.inffus.2019.07.011","article-title":"IFCNN: A general image fusion framework based on convolutional neural network","volume":"54","author":"Zhang","year":"2020","journal-title":"Inf. Fusion"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1016\/j.inffus.2021.12.004","article-title":"Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network","volume":"82","author":"Tang","year":"2022","journal-title":"Inf. Fusion"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Liu, J., Fan, X., Huang, Z., Wu, G., Liu, R., Zhong, W., and Luo, Z. (2022, January 18\u201324). Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.00571"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1034","DOI":"10.1109\/LRA.2025.3636034","article-title":"DQO-MAP: Real-Time Object-Level SLAM with Dual Quadrics and Gaussians","volume":"11","author":"Li","year":"2025","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1261","DOI":"10.1109\/TIP.2020.3043125","article-title":"A bilevel integrated model with data-driven layer ensemble for multi-modality image fusion","volume":"30","author":"Liu","year":"2020","journal-title":"IEEE Trans. Image Process."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Liu, R., Liu, Z., Liu, J., and Fan, X. (2021). Searching a hierarchically aggregated fusion architecture for fast multi-modality image fusion. Proceedings of the 29th ACM International Conference on Multimedia, Association for Computing Machinery.","DOI":"10.1145\/3474085.3475299"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Jia, X., Zhu, C., Li, M., Tang, W., and Zhou, W. (2021, January 11\u201317). LLVIP: A visible-infrared paired dataset for low-light vision. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Montreal, BC, Canada.","DOI":"10.1109\/ICCVW54120.2021.00389"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"0211002","DOI":"10.3788\/AOS201535.0211002","article-title":"Low Contrast Target Polarization Recognition Technology Based on Lifting Wavelet","volume":"35","author":"Zhang","year":"2015","journal-title":"Acta Opt. Sin."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"064205","DOI":"10.7498\/aps.65.064205","article-title":"Research of the Influence of Non-Spherical Ellipsoid Particle Parameter Variation on Polarization Characteristic of Light","volume":"65","author":"Zhang","year":"2016","journal-title":"Acta Phys. Sin."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Fu, Q., Luo, K., Yang, W., Zhan, J., Zhang, S., Shi, H., Li, Y., and Yu, H. (2023). Analysis of Two-Color Infrared Polarization Imaging Characteristics for Target Detection and Recognition. Photonics, 10.","DOI":"10.3390\/photonics10111181"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"9886","DOI":"10.1364\/OE.484233","article-title":"Compressive Space-Dimensional Dual-Coded Hyperspectral Polarimeter (CSDHP) and Interactive Design Method","volume":"31","author":"Wang","year":"2023","journal-title":"Opt. Express"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"2201001","DOI":"10.3788\/AOS202040.2201001","article-title":"Simulation Research on Sky Polarization Characteristics under Complicated Marine Environment","volume":"40","author":"Zhang","year":"2020","journal-title":"Acta Opt. Sin."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Li, H., Xiao, Y., Cheng, C., and Song, X. (2023). SFPFusion: An improved vision transformer combining super feature attention and wavelet-guided pooling for infrared and visible images fusion. Sensors, 23.","DOI":"10.3390\/s23187870"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"101828","DOI":"10.1016\/j.inffus.2023.101828","article-title":"An interactively reinforced paradigm for joint infrared-visible image fusion and saliency object detection","volume":"98","author":"Wang","year":"2023","journal-title":"Inf. Fusion"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"111991","DOI":"10.1016\/j.optlastec.2024.111991","article-title":"DDRF: Dual-branch decomposition and reconstruction architecture for infrared and visible image Fusion","volume":"181","author":"Zhang","year":"2025","journal-title":"Opt. Laser Technol."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"5029714","DOI":"10.1109\/TIM.2024.3450061","article-title":"PFCFuse: A Poolformer and CNN Fusion Network for Infrared-Visible Image Fusion","volume":"73","author":"Hu","year":"2024","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"5000405","DOI":"10.1109\/LGRS.2023.3339214","article-title":"Cross-Modal Feature Fusion and Interaction Strategy for CNN-Transformer-Based Object Detection in Visual and Infrared Remote Sensing Imagery","volume":"21","author":"Nie","year":"2024","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Zhang, C.M., Yuan, C., Luo, Y., and Zhou, X. (2023). MFT: Multi-scale fusion transformer for infrared and visible image fusion. Artificial Neural Networks and Machine Learning\u2014ICANN 2023, Springer Nature.","DOI":"10.1007\/978-3-031-44223-0_39"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Lv, H., Deng, B., and Shi, C. (2021). Research On Multi-Source Image Fusion Target Detection Technology Based on Neural Network. J. Phys. Conf. Ser., 2033.","DOI":"10.1088\/1742-6596\/2033\/1\/012139"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"102839","DOI":"10.1016\/j.inffus.2024.102839","article-title":"Conti-Fuse: A novel continuous decomposition-based fusion framework for infrared and visible Images","volume":"117","author":"Li","year":"2025","journal-title":"Inf. Fusion"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"110223","DOI":"10.1016\/j.patcog.2023.110223","article-title":"Semantic perceptive infrared and visible image fusion Transformer","volume":"149","author":"Yang","year":"2024","journal-title":"Pattern Recognit."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"770","DOI":"10.1109\/TCSVT.2023.3289170","article-title":"Cross-Modal Transformers for Infrared and Visible Image Fusion","volume":"34","author":"Park","year":"2024","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_25","first-page":"5522214","article-title":"Bridging CNN and Transformer with Cross-Attention Fusion Network for Hyperspectral Image Classification","volume":"62","author":"Xu","year":"2024","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_26","first-page":"931","article-title":"FDB-Net: Fusion double branch network combining CNN and transformer for medical image Segmentation","volume":"32","author":"Jiang","year":"2024","journal-title":"J. X-Ray Sci. Technol. Clin. Appl. Diagn. Ther."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"253","DOI":"10.3103\/S0146411622030105","article-title":"Research on Multi-Modal Image Target Recognition Based on Asynchronous Depth Reinforcement Learning","volume":"56","author":"Zhao","year":"2022","journal-title":"Autom. Control Comput. Sci."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Xin, D., Xu, L., Chen, H., Yang, X., and Zhang, R. (2022). A Vehicle Target Detection Method Based on Feature Level Fusion of Infrared and Visible Light Image. 2022 34th Chinese Control and Decision Conference (CCDC), IEEE.","DOI":"10.1109\/CCDC55256.2022.10033899"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"102246","DOI":"10.1016\/j.inffus.2024.102246","article-title":"Improving RGB-infrared object detection with cascade alignment-guided Transformer","volume":"105","author":"Yuan","year":"2024","journal-title":"Inf. Fusion"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"104223","DOI":"10.1016\/j.infrared.2022.104223","article-title":"Infrared and visible image fusion via salient object extraction and Low-light region Enhancement","volume":"124","author":"Liu","year":"2022","journal-title":"Infrared Phys. Technol."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1947","DOI":"10.1177\/03611981241258753","article-title":"CTAFFNet: CNN\u2013Transformer Adaptive Feature Fusion Object Detection Algorithm for Complex Traffic Scenarios","volume":"2679","author":"Dong","year":"2024","journal-title":"Transp. Res. Rec. J. Transp. Res. Board"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Gao, Y., Pei, G., Sheng, M., Sun, Z., Chen, T., and Yao, Y. (2024). Relating CNN-Transformer Fusion Network for Change Detection. arXiv.","DOI":"10.1109\/ICME57554.2024.10687791"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"5280","DOI":"10.1109\/JSTARS.2024.3361507","article-title":"Multiscale Fusion CNN-Transformer Network for High-Resolution Remote Sensing Image Change Detection","volume":"17","author":"Jiang","year":"2024","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"8415","DOI":"10.1007\/s10489-024-05623-7","article-title":"A Transformer-encoder-based multimodal multi-attention fusion network for sentiment Analysis","volume":"54","author":"Liu","year":"2024","journal-title":"Appl. Intell."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"117194","DOI":"10.1016\/j.image.2024.117194","article-title":"Transformer-CNN for small image object Detection","volume":"129","author":"Chen","year":"2024","journal-title":"Signal Process. Image Commun."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"102007","DOI":"10.1016\/j.inffus.2023.102007","article-title":"A reconstruction and convolution operations enabled variant vision transformer with gastroscopic images for automatic locating of polyps in Internet of Medical Things","volume":"101","author":"Qin","year":"2024","journal-title":"Inf. Fusion"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1016\/j.dib.2017.09.038","article-title":"The TNO multiband image data collection","volume":"15","author":"Toet","year":"2017","journal-title":"Data Brief"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"502","DOI":"10.1109\/TPAMI.2020.3012548","article-title":"U2Fusion: A unified unsupervised image fusion network","volume":"44","author":"Xu","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"323","DOI":"10.1016\/j.inffus.2021.06.008","article-title":"Image fusion meets deep learning: A survey and perspective","volume":"76","author":"Zhang","year":"2021","journal-title":"Inf. Fusion"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27\u201330). You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.91"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"11198","DOI":"10.1109\/TCSVT.2024.3418965","article-title":"MMI-Det: Exploring multi-modal integration for visible and infrared object detection","volume":"34","author":"Zeng","year":"2024","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1200","DOI":"10.1109\/JAS.2022.105686","article-title":"SwinFusion: Cross-domain long-range learning for general image fusion via swin transformer","volume":"9","author":"Ma","year":"2022","journal-title":"IEEE\/CAA J. Autom. Sin."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1016\/j.inffus.2022.03.007","article-title":"PIAFusion: A progressive infrared and visible image fusion network based on illumination aware","volume":"83","author":"Tang","year":"2022","journal-title":"Inf. Fusion"},{"key":"ref_44","unstructured":"Jocher, G. (2026, May 16). *ultralytics\/yolov8: V8.1.0\u2014Oriented Bounding Boxes (OBB)*. Available online: https:\/\/github.com\/ultralytics\/ultralytics\/releases\/tag\/v8.1.0."}],"container-title":["Journal of Imaging"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2313-433X\/12\/6\/219\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,24]],"date-time":"2026-05-24T04:24:00Z","timestamp":1779596640000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2313-433X\/12\/6\/219"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,22]]},"references-count":44,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2026,6]]}},"alternative-id":["jimaging12060219"],"URL":"https:\/\/doi.org\/10.3390\/jimaging12060219","relation":{},"ISSN":["2313-433X"],"issn-type":[{"value":"2313-433X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,22]]}}}