{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,8]],"date-time":"2026-03-08T07:00:44Z","timestamp":1772953244866,"version":"3.50.1"},"reference-count":49,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2022,5,11]],"date-time":"2022-05-11T00:00:00Z","timestamp":1652227200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Pixel-level image fusion is an effective way to fully exploit the rich texture information of visible images and the salient target characteristics of infrared images. With the development of deep learning technology in recent years, the image fusion algorithm based on this method has also achieved great success. However, owing to the lack of sufficient and reliable paired data and a nonexistent ideal fusion result as supervision, it is difficult to design a precise network training mode. Moreover, the manual fusion strategy has difficulty ensuring the full use of information, which easily causes redundancy and omittance. To solve the above problems, this paper proposes a multi-stage visible and infrared image fusion network based on an attention mechanism (MSFAM). Our method stabilizes the training process through multi-stage training and enhances features by the learning attention fusion block. To improve the network effect, we further design a Semantic Constraint module and Push\u2013Pull loss function for the fusion task. Compared with several recently used methods, the qualitative comparison intuitively shows more beautiful and natural fusion results by our model with a stronger applicability. For quantitative experiments, MSFAM achieves the best results in three of the six frequently used metrics in fusion tasks, while other methods only obtain good scores on a single metric or a few metrics. Besides, a commonly used high-level semantic task, i.e., object detection, is used to prove its greater benefits for downstream tasks compared with singlelight images and fusion results by existing methods. All these experiments prove the superiority and effectiveness of our algorithm.<\/jats:p>","DOI":"10.3390\/s22103651","type":"journal-article","created":{"date-parts":[[2022,5,12]],"date-time":"2022-05-12T23:08:36Z","timestamp":1652396916000},"page":"3651","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["A Multi-Stage Visible and Infrared Image Fusion Network Based on Attention Mechanism"],"prefix":"10.3390","volume":"22","author":[{"given":"Xin","family":"Zheng","sequence":"first","affiliation":[{"name":"Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Qiyong","family":"Yang","sequence":"additional","affiliation":[{"name":"Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pengbo","family":"Si","sequence":"additional","affiliation":[{"name":"Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Qiang","family":"Wu","sequence":"additional","affiliation":[{"name":"Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,5,11]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1007\/s11263-006-6655-0","article-title":"Multiscale fusion of visible and thermal IR images for illumination-invariant face recognition","volume":"71","author":"Kong","year":"2007","journal-title":"Int. J. Comput. Vis."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Bavirisetti, D.P., Xiao, G., and Liu, G. (2017, January 10\u201313). Multi-sensor image fusion based on fourth order partial differential equations. Proceedings of the 2017 20th International Conference on Information Fusion (Fusion), Xi\u2019an, China.","DOI":"10.23919\/ICIF.2017.8009719"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1016\/j.inffus.2017.05.006","article-title":"Sparse representation based multi-sensor image fusion for multi-focus and multi-modality images: A review","volume":"40","author":"Zhang","year":"2018","journal-title":"Inf. Fusion"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1016\/j.inffus.2018.02.004","article-title":"Infrared and visible image fusion methods and applications: A survey","volume":"45","author":"Ma","year":"2019","journal-title":"Inf. Fusion"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Zhang, X., Ye, P., and Xiao, G. (2020, January 14\u201319). VIFB: A Visible and Infrared Image Fusion Benchmark. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, WA, USA.","DOI":"10.1109\/CVPRW50498.2020.00060"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1016\/j.inffus.2014.09.004","article-title":"A general framework for image fusion based on multi-scale transform and sparse representation","volume":"24","author":"Liu","year":"2015","journal-title":"Inf. Fusion"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1016\/j.infrared.2016.01.009","article-title":"Two-scale image fusion of visible and infrared images using saliency detection","volume":"76","author":"Bavirisetti","year":"2016","journal-title":"Infrared Phys. Technol."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"9645","DOI":"10.1109\/TIM.2020.3005230","article-title":"NestFuse: An Infrared and Visible Image Fusion Architecture based on Nest Connection and Spatial\/Channel Attention Models","volume":"69","author":"Li","year":"2020","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_9","first-page":"2234","article-title":"Improved techniques for training gans","volume":"29","author":"Salimans","year":"2016","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"110","DOI":"10.1016\/j.inffus.2021.02.019","article-title":"Image fusion based on generative adversarial network consistent with perception","volume":"72","author":"Fu","year":"2021","journal-title":"Inf. Fusion"},{"key":"ref_11","first-page":"1","article-title":"STDFusionNet: An Infrared and Visible Image Fusion Network Based on Salient Target Detection","volume":"70","author":"Ma","year":"2021","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_12","first-page":"1","article-title":"GANMcC: A Generative Adversarial Network With Multiclassification Constraints for Infrared and Visible Image Fusion","volume":"70","author":"Ma","year":"2020","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_13","unstructured":"Toet, A. (2022, May 08). TNO Image Fusion Dataset. Available online: https:\/\/figshare.com\/articles\/dataset\/TNO_Image_Fusion_Dataset\/1008029."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Xu, H., Ma, J., Le, Z., Jiang, J., and Guo, X. (2020, January 7\u201312). FusionDN: A Unified Densely Connected Network for Image Fusion. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.","DOI":"10.1609\/aaai.v34i07.6936"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"502","DOI":"10.1109\/TPAMI.2020.3012548","article-title":"U2Fusion: A Unified Unsupervised Image Fusion Network","volume":"44","author":"Xu","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"103039","DOI":"10.1016\/j.infrared.2019.103039","article-title":"Infrared and Visible Image Fusion with ResNet and zero-phase component analysis","volume":"102","author":"Li","year":"2019","journal-title":"Infrared Phys. Technol."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"34685","DOI":"10.1007\/s11042-020-09301-x","article-title":"Unsupervised densely attention network for infrared and visible image fusion","volume":"79","author":"Li","year":"2020","journal-title":"Multimed. Tools Appl."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Fu, Y., and Wu, X.J. (2021, January 10\u201315). A Dual-branch Network for Infrared and Visible Image Fusion. Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy.","DOI":"10.1109\/ICPR48806.2021.9412293"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Zhao, Z., Xu, S., Zhang, C., Liu, J., Li, P., and Zhang, J. (2021, January 7\u201315). DIDFuse: Deep Image Decomposition for Infrared and Visible Image Fusion. Proceedings of theTwenty-Ninth International Joint Conference on Artificial Intelligence and Seventeenth Pacific Rim International Conference on Artificial Intelligence, Yokohama, Japan.","DOI":"10.24963\/ijcai.2020\/135"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"2614","DOI":"10.1109\/TIP.2018.2887342","article-title":"DenseFuse: A Fusion Approach to Infrared and Visible Images","volume":"28","author":"Li","year":"2018","journal-title":"IEEE Trans. Image Process. A Publ. IEEE Signal Processing Soc."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TIM.2020.3022438","article-title":"SEDRFuse: A Symmetric Encoder-Decoder with Residual Block Network for Infrared and Visible Image Fusion","volume":"70","author":"Jian","year":"2020","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"4980","DOI":"10.1109\/TIP.2020.2977573","article-title":"DDcGAN: A Dual-Discriminator Conditional Generative Adversarial Network for Multi-Resolution Image Fusion","volume":"29","author":"Ma","year":"2020","journal-title":"IEEE Trans. Image Process."},{"key":"ref_23","first-page":"3739040","article-title":"The Fusion of Unmatched Infrared and Visible Images Based on Generative Adversarial Networks","volume":"2020","author":"Zhao","year":"2020","journal-title":"Math. Probl. Eng."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"4771","DOI":"10.1109\/TCSVT.2021.3054584","article-title":"Infrared and Visible Image Fusion via Texture Conditional Generative Adversarial Network","volume":"31","author":"Yang","year":"2021","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"206445","DOI":"10.1109\/ACCESS.2020.3037770","article-title":"Infrared and Visible Image Fusion Using a Deep Unsupervised Framework with Perceptual Loss","volume":"8","author":"Xu","year":"2020","journal-title":"IEEE Access"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"509","DOI":"10.1177\/1073858413514136","article-title":"Bottom-up and top-down attention: Different processes and overlapping neural systems","volume":"20","author":"Katsuki","year":"2014","journal-title":"Neuroscientist"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"194","DOI":"10.1038\/35058500","article-title":"Computational modelling of visual attention","volume":"2","author":"Itti","year":"2001","journal-title":"Nat. Rev. Neurosci."},{"key":"ref_28","unstructured":"Jie, H., Li, S., and Gang, S. (2017, January 21\u201326). Squeeze-and-Excitation Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Woo, S., Park, J., Lee, J.Y., and Kweon, I.S. (2018). CBAM: Convolutional Block Attention Module, Springer.","DOI":"10.1007\/978-3-030-01234-2_1"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"15051","DOI":"10.1007\/s11042-015-2512-x","article-title":"Semi-supervised vision-based maritime surveillance system using fused visual attention maps","volume":"75","author":"Makantasis","year":"2016","journal-title":"Multimed. Tools Appl."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"9325","DOI":"10.1109\/ACCESS.2020.2964540","article-title":"Attention mask R-CNN for ship detection and segmentation from remote sensing images","volume":"8","author":"Nie","year":"2020","journal-title":"IEEE Access"},{"key":"ref_32","unstructured":"Woo, S., Park, J., Lee, J.Y., and Kweon, I.S. (2018, January 18\u201323). Non-local neural networks. Proceedings of the IEEE Conference On Computer Vision and Pattern Recognition, Salt Lake City, UT, USA."},{"key":"ref_33","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, Morgan Kaufmann Publishers."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Liu, D., Wen, B., Liu, X., Wang, Z., and Huang, T.S. (2017). When image denoising meets high-level vision tasks: A deep learning approach. arXiv.","DOI":"10.24963\/ijcai.2018\/117"},{"key":"ref_35","unstructured":"Wang, S., Wen, B., Wu, J., Tao, D., and Wang, Z. (2019). Segmentation-aware image denoising without knowing true segmentation. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Li, B., Peng, X., Wang, Z., Xu, J., and Feng, D. (2017, January 22\u201329). AOD-Net: All-in-One Dehazing Network. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.511"},{"key":"ref_37","unstructured":"Zhao, M., Cheng, L., Yang, X., Feng, P., Liu, L., and Wu, N. (2019). Tbc-net: A real-time detector for infrared small target detection using semantic constraint. arXiv."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Chen, L., Lu, X., Zhang, J., Chu, X., and Chen, C. (2021, January 20\u201325). HINet: Half instance normalization network for image restoration. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPRW53098.2021.00027"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1109\/TCI.2016.2644865","article-title":"Loss Functions for Image Restoration with Neural Networks","volume":"3","author":"Zhao","year":"2017","journal-title":"IEEE Trans. Comput. Imaging"},{"key":"ref_40","first-page":"181","article-title":"Objective image fusion performance measure","volume":"56","author":"Xydeas","year":"2000","journal-title":"Mil. Technol. Cour."},{"key":"ref_41","unstructured":"Zhou, W., Simoncelli, E.P., and Bovik, A.C. (2003, January 9\u201312). Multiscale structural similarity for image quality assessment. Proceedings of the he Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, Pacific Grove, CA, USA."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"023522","DOI":"10.1117\/1.2945910","article-title":"Assessment of image fusion procedures using entropy, image quality, and multispectral classification","volume":"2","author":"Roberts","year":"2008","journal-title":"J. Appl. Remote Sens."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1890","DOI":"10.1016\/j.aeue.2015.09.004","article-title":"A new image quality metric for image fusion: The sum of the correlations of differences","volume":"69","author":"Aslantas","year":"2015","journal-title":"AEU Int. J. Electron. Commun."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"313","DOI":"10.1049\/el:20020212","article-title":"Information measure for performance of image fusion","volume":"38","author":"Qu","year":"2002","journal-title":"Electron. Lett."},{"key":"ref_45","first-page":"484","article-title":"Image fusion and image quality assessment of fused images","volume":"4","author":"Deshmukh","year":"2010","journal-title":"Int. J. Image Process."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1016\/j.inffus.2015.11.003","article-title":"Perceptual fusion of infrared and visible images through a hybrid multi-scale decomposition with Gaussian and bilateral filters","volume":"30","author":"Zhou","year":"2016","journal-title":"Inf. Fusion"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Zhang, H., Xu, H., Xiao, Y., Guo, X., and Ma, J. (2020, January 7\u201312). Rethinking the Image Fusion: A Fast Unified Image Fusion Network based on Proportional Maintenance of Gradient and Intensity. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.","DOI":"10.1609\/aaai.v34i07.6975"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Hwang, S., Park, J., Kim, N., Choi, Y., and So Kweon, I. (2015, January 7\u201312). Multispectral pedestrian detection: Benchmark dataset and baseline. Proceedings of the IEEE Conference On Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298706"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Li, X., Wang, W., Hu, X., Li, J., Tang, J., and Yang, J. (2021, January 20\u201325). Generalized focal loss v2: Learning reliable localization quality estimation for dense object detection. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.01146"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/10\/3651\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:09:07Z","timestamp":1760137747000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/10\/3651"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,11]]},"references-count":49,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2022,5]]}},"alternative-id":["s22103651"],"URL":"https:\/\/doi.org\/10.3390\/s22103651","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,5,11]]}}}