{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,24]],"date-time":"2025-12-24T18:19:43Z","timestamp":1766600383886,"version":"build-2065373602"},"reference-count":50,"publisher":"MDPI AG","issue":"14","license":[{"start":{"date-parts":[[2023,7,12]],"date-time":"2023-07-12T00:00:00Z","timestamp":1689120000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The challenging issues in infrared and visible image fusion (IVIF) are extracting and fusing as much useful information as possible contained in the source images, namely, the rich textures in visible images and the significant contrast in infrared images. Existing fusion methods cannot address this problem well due to the handcrafted fusion operations and the extraction of features only from a single scale. In this work, we solve the problems of insufficient information extraction and fusion from another perspective to overcome the difficulties in lacking textures and unhighlighted targets in fused images. We propose a multi-scale feature extraction (MFE) and joint attention fusion (JAF) based end-to-end method using a generative adversarial network (MJ-GAN) framework for the aim of IVIF. The MFE modules are embedded in the two-stream structure-based generator in a densely connected manner to comprehensively extract multi-grained deep features from the source image pairs and reuse them during reconstruction. Moreover, an improved self-attention structure is introduced into the MFEs to enhance the pertinence among multi-grained features. The merging procedure for salient and important features is conducted via the JAF network in a feature recalibration manner, which also produces the fused image in a reasonable manner. Eventually, we can reconstruct a primary fused image with the major infrared radiometric information and a small amount of visible texture information via a single decoder network. The dual discriminator with strong discriminative power can add more texture and contrast information to the final fused image. Extensive experiments on four publicly available datasets show that the proposed method ultimately achieves phenomenal performance in both visual quality and quantitative assessment compared with nine leading algorithms.<\/jats:p>","DOI":"10.3390\/s23146322","type":"journal-article","created":{"date-parts":[[2023,7,13]],"date-time":"2023-07-13T01:58:11Z","timestamp":1689213491000},"page":"6322","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["MJ-GAN: Generative Adversarial Network with Multi-Grained Feature Extraction and Joint Attention Fusion for Infrared and Visible Image Fusion"],"prefix":"10.3390","volume":"23","author":[{"given":"Danqing","family":"Yang","sequence":"first","affiliation":[{"name":"School of Optoelectronic Engineering, Xidian University, Xi\u2019an 710071, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiaorui","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Optoelectronic Engineering, Xidian University, Xi\u2019an 710071, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Naibo","family":"Zhu","sequence":"additional","affiliation":[{"name":"Research Institute of System Engineering, PLA Academy of Military Science, Beijing 100091, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shuang","family":"Li","sequence":"additional","affiliation":[{"name":"Research Institute of System Engineering, PLA Academy of Military Science, Beijing 100091, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Na","family":"Hou","sequence":"additional","affiliation":[{"name":"Research Institute of System Engineering, PLA Academy of Military Science, Beijing 100091, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,7,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1016\/j.inffus.2018.02.004","article-title":"Infrared and visible image fusion methods and applications: A survey","volume":"45","author":"Ma","year":"2019","journal-title":"Inf. Fusion"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"635","DOI":"10.1109\/TMM.2021.3129609","article-title":"Semantic-Supervised Infrared and Visible Image Fusion Via a Dual-Discriminator Generative Adversarial Network","volume":"25","author":"Zhou","year":"2021","journal-title":"IEEE Trans. Multimed."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1134","DOI":"10.1109\/TCI.2021.3119954","article-title":"GAN-FM: Infrared and Visible Image Fusion Using GAN with Full-Scale Skip Connection and Dual Markovian Discriminators","volume":"7","author":"Zhang","year":"2021","journal-title":"IEEE Trans. Comput. Imaging"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1261","DOI":"10.1109\/TIP.2020.3043125","article-title":"A Bilevel Integrated Model with Data-Driven Layer Ensemble for Multi-Modality Image Fusion","volume":"30","author":"Liu","year":"2021","journal-title":"IEEE Trans. Image Process."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1016\/j.inffus.2016.05.004","article-title":"Pixel-level image fusion: A survey of the state of the art","volume":"33","author":"Li","year":"2017","journal-title":"Inf. Fusion"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"478","DOI":"10.1016\/j.infrared.2017.07.010","article-title":"A survey of infrared and visual image fusion methods","volume":"85","author":"Jin","year":"2017","journal-title":"Infrared Phys. Technol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1016\/j.ins.2019.08.066","article-title":"Infrared and visible image fusion based on target-enhanced multiscale transform decomposition","volume":"508","author":"Chen","year":"2020","journal-title":"Inf. Sci."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1016\/j.inffus.2017.05.006","article-title":"Sparse representation based multi-sensor image fusion for multi-focus and multi-modality images: A review","volume":"40","author":"Zhang","year":"2018","journal-title":"Inf. Fusion"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1016\/j.infrared.2013.11.008","article-title":"Infrared image enhancement through saliency feature analysis based on multi-scale decom-position","volume":"62","author":"Zhao","year":"2014","journal-title":"Infrared Phys. Technol."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1016\/j.infrared.2014.07.019","article-title":"Adaptive fusion method of visible light and infrared images based on non-subsampled shearlet transform and fast non-negative matrix factorization","volume":"67","author":"Kong","year":"2014","journal-title":"Infrared Phys. Technol."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1016\/j.inffus.2014.09.004","article-title":"A general framework for image fusion based on multi-scale transform and sparse representation","volume":"24","author":"Liu","year":"2015","journal-title":"Inf. Fusion"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1016\/j.infrared.2017.01.012","article-title":"Fusion of visible and infrared images using global entropy and gradient constrained regularization","volume":"81","author":"Zhao","year":"2017","journal-title":"Infrared Phys. Technol."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"20286","DOI":"10.1109\/ACCESS.2017.2758644","article-title":"A novel multi-focus image fusion method based on stationary wavelet transform and local features of fuzzy sets","volume":"5","author":"Jiang","year":"2017","journal-title":"IEEE Access"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1113","DOI":"10.1016\/j.asoc.2017.02.035","article-title":"A multi-faceted adaptive image fusion algorithm using a multi-wavelet-based matching measure in the PCNN domain","volume":"61","author":"Wang","year":"2017","journal-title":"Appl. Soft Comput."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1850018","DOI":"10.1142\/S0219691318500182","article-title":"Infrared and visible image fusion with convolutional neural networks","volume":"16","author":"Liu","year":"2018","journal-title":"Int. J. Wavelets Multi."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Li, H., Wu, X.-J., and Kittler, J. (2018, January 20\u201324). Infrared and Visible Image Fusion using a Deep Learning Framework. Proceedings of the 2018 24th International Conference on Pattern Recognition (ICPR), Beijing, China.","DOI":"10.1109\/ICPR.2018.8546006"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1882","DOI":"10.1109\/LSP.2016.2618776","article-title":"Image Fusion with Convolutional Sparse Representation","volume":"23","author":"Liu","year":"2016","journal-title":"IEEE Signal Process. Lett."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"2614","DOI":"10.1109\/TIP.2018.2887342","article-title":"DenseFuse: A Fusion Approach to Infrared and Visible Images","volume":"28","author":"Li","year":"2018","journal-title":"IEEE Trans. Image Process."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1016\/j.neucom.2021.05.034","article-title":"Two-stream network for infrared and visible images fusion","volume":"460","author":"Liu","year":"2021","journal-title":"Neurocomputing"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Prabhakar, K.R., Srikar, V.S., and Babu, R.V. (2017, January 22\u201329). DeepFuse: A Deep Unsupervised Approach for Exposure Fusion with Extreme Exposure Image Pairs. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.505"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"128","DOI":"10.1016\/j.inffus.2020.11.009","article-title":"RXDNFuse: A aggregated residual dense network for infrared and visible image fusion","volume":"69","author":"Long","year":"2021","journal-title":"Inf. Fusion"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1016\/j.inffus.2018.09.004","article-title":"FusionGAN: A generative adversarial network for infrared and visible image fusion","volume":"48","author":"Ma","year":"2019","journal-title":"Inf. Fusion"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1016\/j.ins.2020.04.035","article-title":"Infrared and visible image fusion using dual discriminators generative adversarial networks with Wasserstein distance","volume":"529","author":"Li","year":"2020","journal-title":"Inf. Sci."},{"key":"ref_24","first-page":"3038013","article-title":"GANMcC: A Generative Adversarial Network with Multiclassification Constraints for Infrared and Visible Image Fusion","volume":"70","author":"Ma","year":"2020","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1890","DOI":"10.1016\/j.aeue.2015.09.004","article-title":"A new image quality metric for image fusion: The sum of the correlations of differences","volume":"69","author":"Aslantas","year":"2015","journal-title":"AEU\u2014Int. J. Electron. Commun."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"3075747","DOI":"10.1109\/TIM.2021.3075747","article-title":"STDFusionNet: An Infrared and Visible Image Fusion Network Based on Salient Target Detection","volume":"70","author":"Ma","year":"2021","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_27","unstructured":"Radford, A., Metz, L., and Chintala, S. (2016). Unsupervised representation learning with deep convolutional generative adversarial net-works. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"7458","DOI":"10.1109\/JSEN.2019.2921803","article-title":"Coupled GAN with Relativistic Discriminators for Infrared and Visible Images Fusion","volume":"21","author":"Li","year":"2019","journal-title":"IEEE Sens. J."},{"key":"ref_29","first-page":"5002412","article-title":"Multi-grained attention network for infrared and visible image fusion","volume":"70","author":"Li","year":"2020","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1383","DOI":"10.1109\/TMM.2020.2997127","article-title":"AttentionFGAN: Infrared and Visible Image Fusion Using Attention-Based Generative Adversarial Networks","volume":"23","author":"Li","year":"2020","journal-title":"IEEE Trans. Multimed."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21\u201326). Densely connected convolutional networks. Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.243"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Fu, Y., and Wu, X.-J. (2021, January 10\u201315). A Dual-Branch Network for Infrared and Visible Image Fusion. Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy.","DOI":"10.1109\/ICPR48806.2021.9412293"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"824","DOI":"10.1109\/TCI.2021.3100986","article-title":"CSF: Classification saliency-based rule for visible and infrared image fusion","volume":"7","author":"Xu","year":"2021","journal-title":"IEEE Trans. Comput. Imaging"},{"key":"ref_34","unstructured":"Lu, J., Yang, J., Batra, D., and Parikh, D. (2016). Hierarchical question-image co-attention for visual question answering. NIPS, 9."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"382","DOI":"10.1109\/TAFFC.2020.3025777","article-title":"Eeg-based emotion recognition via channel-wise attention and self-attention","volume":"14","author":"Tao","year":"2020","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Wang, X., and Tang, X. (2017, January 21\u201326). Residual Attention Network for Image Classification. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.683"},{"key":"ref_37","first-page":"12797","article-title":"Rethinking the Image Fusion: A Fast Unified Image Fusion Network based on Proportional Maintenance of Gradient and Intensity","volume":"34","author":"Zhang","year":"2020","journal-title":"Proc. Conf. AAAI Artif. Intell."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1489","DOI":"10.1109\/TPAMI.2022.3164083","article-title":"Contextual Transformer Networks for Visual Recognition","volume":"45","author":"Li","year":"2022","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"2011","DOI":"10.1109\/TPAMI.2019.2913372","article-title":"Squeeze-and-Excitation Networks","volume":"42","author":"Hu","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"6133","DOI":"10.1007\/s00521-020-05387-4","article-title":"GANFuse: A novel multi-exposure image fusion method based on generative adversarial networks","volume":"33","author":"Yang","year":"2020","journal-title":"Neural Comput. Appl."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1125","DOI":"10.1007\/s11760-012-0361-x","article-title":"Multifocus and multispectral image fusion based on pixel significance using discrete cosine harmonic wavelet transform","volume":"7","author":"Kumar","year":"2013","journal-title":"Signal Image Video Process."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"479","DOI":"10.14429\/dsj.61.705","article-title":"Image Fusion Technique using Multi-resolution Singular Value Decomposition","volume":"61","author":"Naidu","year":"2011","journal-title":"Def. Sci. J."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1016\/j.inffus.2013.11.005","article-title":"Multi-scale weighted gradient-based fusion for multi-focus images","volume":"20","author":"Zhou","year":"2014","journal-title":"Inf. Fusion"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Ma, W., Wang, K., and Li, J. (2023). Infrared and visible image fusion technology and application: A review. Sensors, 23.","DOI":"10.3390\/s23020599"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"313","DOI":"10.1049\/el:20020212","article-title":"Information measure for performance of image fusion","volume":"38","author":"Qu","year":"2002","journal-title":"Electron. Lett."},{"key":"ref_46","first-page":"484","article-title":"Image fusion and image quality assessment of fused images","volume":"4","author":"Deshmukh","year":"2010","journal-title":"Int. J. Image Process."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"B125","DOI":"10.1364\/JOSAA.24.00B125","article-title":"Selection of image fusion quality measures: Objective, subjective, and metric assessment","volume":"24","author":"Dixon","year":"2007","journal-title":"J. Opt. Soc. Am. A"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"2959","DOI":"10.1109\/26.477498","article-title":"Image quality measures and their performance","volume":"43","author":"Eskicioglu","year":"1995","journal-title":"IEEE Trans. Commun."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"600","DOI":"10.1109\/TIP.2003.819861","article-title":"Image Quality Assessment: From Error Visibility to Structural Similarity","volume":"13","author":"Wang","year":"2004","journal-title":"IEEE Trans. Image Process."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"023522","DOI":"10.1117\/1.2945910","article-title":"Assessment of image fusion procedures using entropy, image quality, and multispectral classification","volume":"2","author":"Roberts","year":"2008","journal-title":"J. Appl. Remote Sens."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/14\/6322\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:11:08Z","timestamp":1760127068000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/14\/6322"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,12]]},"references-count":50,"journal-issue":{"issue":"14","published-online":{"date-parts":[[2023,7]]}},"alternative-id":["s23146322"],"URL":"https:\/\/doi.org\/10.3390\/s23146322","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2023,7,12]]}}}