{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T02:07:14Z","timestamp":1760148434482,"version":"build-2065373602"},"reference-count":65,"publisher":"MDPI AG","issue":"9","license":[{"start":{"date-parts":[[2023,5,6]],"date-time":"2023-05-06T00:00:00Z","timestamp":1683331200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62273353"],"award-info":[{"award-number":["62273353"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Image fusion is the process of combining multiple input images from single or multiple imaging modalities into a fused image, which is expected to be more informative for human or machine perception as compared to any of the input images. In this paper, we propose a novel method based on deep learning for fusing infrared images and visible images, named the local binary pattern (LBP)-based proportional input generative adversarial network (LPGAN). In the image fusion task, the preservation of structural similarity and image gradient information is contradictory, and it is difficult for both to achieve good performance at the same time. To solve this problem, we innovatively introduce LBP into GANs, enabling the network to have stronger texture feature extraction and utilization capabilities, as well as anti-interference capabilities. In the feature extraction stage, we introduce a pseudo-Siamese network for the generator to extract the detailed features and the contrast features. At the same time, considering the characteristic distribution of different modal images, we propose a 1:4 scale input mode. Extensive experiments on the publicly available TNO dataset and CVC14 dataset show that the proposed method achieves the state-of-the-art performance. We also test the universality of LPGAN by fusing RGB and infrared images on the RoadScene dataset and medical images. In addition, LPGAN is applied to multi-spectral remote sensing image fusion. Both qualitative and quantitative experiments demonstrate that our LPGAN can not only achieve good structural similarity, but also retain richly detailed information.<\/jats:p>","DOI":"10.3390\/rs15092440","type":"journal-article","created":{"date-parts":[[2023,5,8]],"date-time":"2023-05-08T02:03:31Z","timestamp":1683511411000},"page":"2440","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["LPGAN: A LBP-Based Proportional Input Generative Adversarial Network for Image Fusion"],"prefix":"10.3390","volume":"15","author":[{"given":"Dongxu","family":"Yang","sequence":"first","affiliation":[{"name":"College of Intelligence Science and Technology, National University of Defense Technology, Changsha 410073, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3631-7929","authenticated-orcid":false,"given":"Yongbin","family":"Zheng","sequence":"additional","affiliation":[{"name":"College of Intelligence Science and Technology, National University of Defense Technology, Changsha 410073, China"}]},{"given":"Wanying","family":"Xu","sequence":"additional","affiliation":[{"name":"College of Intelligence Science and Technology, National University of Defense Technology, Changsha 410073, China"}]},{"given":"Peng","family":"Sun","sequence":"additional","affiliation":[{"name":"College of Intelligence Science and Technology, National University of Defense Technology, Changsha 410073, China"}]},{"given":"Di","family":"Zhu","sequence":"additional","affiliation":[{"name":"College of Intelligence Science and Technology, National University of Defense Technology, Changsha 410073, China"}]}],"member":"1968","published-online":{"date-parts":[[2023,5,6]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1016\/j.inffus.2016.05.004","article-title":"Pixel-level image fusion: A survey of the state of the art","volume":"33","author":"Li","year":"2017","journal-title":"Inf. Fusion"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"2864","DOI":"10.1109\/TIP.2013.2244222","article-title":"Image fusion with guided filtering","volume":"22","author":"Li","year":"2013","journal-title":"IEEE Trans. Image Process."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Yang, J., Zhao, Y., and Chan, J.C.W. (2018). Hyperspectral and Multispectral Image Fusion via Deep Two-Branches Convolutional Neural Network. Remote Sens., 10.","DOI":"10.3390\/rs10050800"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Sun, K., and Tian, Y. (2023). DBFNet: A Dual-Branch Fusion Network for Underwater Image Enhancement. Remote Sens., 15.","DOI":"10.3390\/rs15051195"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1808","DOI":"10.1109\/JSTARS.2015.2489838","article-title":"Developing a Spectral-Based Strategy for Urban Object Detection From Airborne Hyperspectral TIR and Visible Data","volume":"9","author":"Eslami","year":"2016","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Wang, J., Li, L., Liu, Y., Hu, J., Xiao, X., and Liu, B. (2023). AI-TFNet: Active Inference Transfer Convolutional Fusion Network for Hyperspectral Image Classification. Remote Sens., 15.","DOI":"10.3390\/rs15051292"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1391","DOI":"10.1109\/TGRS.2005.846874","article-title":"A comparative analysis of image fusion methods","volume":"43","author":"Wang","year":"2005","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_8","first-page":"5517218","article-title":"Fusion of Hyperspectral and Multispectral Images Accounting for Localized Inter-image Changes","volume":"60","author":"Fu","year":"2021","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1016\/j.inffus.2013.12.002","article-title":"Medical image fusion: A survey of the state of the art","volume":"19","author":"James","year":"2014","journal-title":"Inf. Fusion"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1016\/j.inffus.2016.03.003","article-title":"A review of remote sensing image fusion methods","volume":"32","author":"Ghassemian","year":"2016","journal-title":"Inf. Fusion"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1016\/j.inffus.2018.02.004","article-title":"Infrared and visible image fusion methods and applications: A survey","volume":"45","author":"Ma","year":"2019","journal-title":"Inf. Fusion"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"2706","DOI":"10.1109\/TMM.2017.2711422","article-title":"An adaptive fusion algorithm for visible and infrared videos based on entropy and the cumulative distribution of gray levels","volume":"19","author":"Hu","year":"2017","journal-title":"IEEE Trans. Multimed."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"015011","DOI":"10.1117\/1.JRS.11.015011","article-title":"Infrared and visible image fusion based on target extraction in the nonsubsampled contourlet transform domain","volume":"11","author":"He","year":"2017","journal-title":"J. Appl. Remote. Sens."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1650024","DOI":"10.1142\/S0219691316500247","article-title":"Efficient image fusion with approximate sparse representation","volume":"14","author":"Bin","year":"2016","journal-title":"Int. J. Wavelets Multiresolut. Inf. Process."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1016\/j.inffus.2017.05.006","article-title":"Sparse representation based multi-sensor image fusion for multi-focus and multi-modality images: A review","volume":"40","author":"Zhang","year":"2018","journal-title":"Inf. Fusion"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1007\/s12596-013-0148-7","article-title":"Hybrid DDCT-PCA based multi sensor image fusion","volume":"43","author":"Naidu","year":"2014","journal-title":"J. Opt."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1016\/j.infrared.2017.02.005","article-title":"Infrared and visible image fusion based on visual saliency map and weighted least square optimization","volume":"82","author":"Ma","year":"2017","journal-title":"Infrared Phys. Technol."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TIM.2020.2986875","article-title":"Infrared and visible image fusion using visual saliency sparse representation and detail injection model","volume":"70","author":"Yang","year":"2020","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"182","DOI":"10.1016\/j.neucom.2016.11.051","article-title":"A novel infrared and visible image fusion algorithm based on shift-invariant dual-tree complex shearlet transform and sparse representation","volume":"226","author":"Yin","year":"2017","journal-title":"Neurocomputing"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"6346","DOI":"10.3390\/rs5126346","article-title":"An Improved Image Fusion Approach Based on Enhanced Spatial and Temporal the Adaptive Reflectance Fusion Model","volume":"5","author":"Fu","year":"2013","journal-title":"Remote Sens."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1016\/j.inffus.2016.02.001","article-title":"Infrared and visible image fusion via gradient transfer and total variation minimization","volume":"31","author":"Ma","year":"2016","journal-title":"Inf. Fusion"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1016\/j.neucom.2016.03.009","article-title":"Infrared and visible image fusion using total variation model","volume":"202","author":"Ma","year":"2016","journal-title":"Neurocomputing"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1016\/j.infrared.2015.01.002","article-title":"A fusion algorithm for infrared and visible images based on adaptive dual-channel unit-linking PCNN in NSCT domain","volume":"69","author":"Xiang","year":"2015","journal-title":"Infrared Phys. Technol."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"158","DOI":"10.1016\/j.inffus.2017.10.007","article-title":"Deep learning for pixel-level image fusion: Recent advances and future prospects","volume":"42","author":"Liu","year":"2018","journal-title":"Inf. Fusion"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Xu, F., Liu, J., Song, Y., Sun, H., and Wang, X. (2022). Multi-Exposure Image Fusion Techniques: A Comprehensive Review. Remote Sens., 14.","DOI":"10.3390\/rs14030771"},{"key":"ref_26","unstructured":"Yang, D., Zheng, Y., Xu, W., Sun, P., and Zhu, D. (2022). International Conference on Guidance, Navigation and Control, Springer."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"2614","DOI":"10.1109\/TIP.2018.2887342","article-title":"DenseFuse: A fusion approach to infrared and visible images","volume":"28","author":"Li","year":"2018","journal-title":"IEEE Trans. Image Process."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1016\/j.inffus.2018.09.004","article-title":"FusionGAN: A generative adversarial network for infrared and visible image fusion","volume":"48","author":"Ma","year":"2019","journal-title":"Inf. Fusion"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"4980","DOI":"10.1109\/TIP.2020.2977573","article-title":"DDcGAN: A dual-discriminator conditional generative adversarial network for multi-resolution image fusion","volume":"29","author":"Ma","year":"2020","journal-title":"IEEE Trans. Image Process."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"502","DOI":"10.1109\/TPAMI.2020.3012548","article-title":"U2Fusion: A unified unsupervised image fusion network","volume":"44","author":"Xu","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_31","unstructured":"Ojala, T., Pietikainen, M., and Harwood, D. (1994, January 9\u201313). Performance evaluation of texture measures with classification based on Kullback discrimination of distributions. Proceedings of the 12th International Conference on Pattern Recognition, Jerusalem, Israel."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Zhang, H., Xu, H., Xiao, Y., Guo, X., and Ma, J. (2020, January 7\u201312). Rethinking the Image Fusion: A Fast Unified Image Fusion Network based on Proportional Maintenance of Gradient and Intensity. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.","DOI":"10.1609\/aaai.v34i07.6975"},{"key":"ref_33","first-page":"5005014","article-title":"GANMcC: A Generative Adversarial Network With Multiclassification Constraints for Infrared and Visible Image Fusion","volume":"70","author":"Ma","year":"2020","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Gonz\u00e1lez, A., Fang, Z., Socarras, Y., Serrat, J., V\u00e1zquez, D., Xu, J., and L\u00f3pez, A.M. (2016). Pedestrian detection at day\/night time with visible and FIR cameras: A comparison. Sensors, 16.","DOI":"10.3390\/s16060820"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1016\/j.inffus.2019.07.005","article-title":"Infrared and visible image fusion via detail preserving adversarial learning","volume":"54","author":"Ma","year":"2020","journal-title":"Inf. Fusion"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1882","DOI":"10.1109\/LSP.2016.2618776","article-title":"Image fusion with convolutional sparse representation","volume":"23","author":"Liu","year":"2016","journal-title":"IEEE Signal Process. Lett."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1016\/j.inffus.2016.12.001","article-title":"Multi-focus image fusion with a deep convolutional neural network","volume":"36","author":"Liu","year":"2017","journal-title":"Inf. Fusion"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Li, H., Wu, X.J., and Kittler, J. (2018, January 20\u201324). Infrared and visible image fusion using a deep learning framework. Proceedings of the 2018 24th International Conference on Pattern Recognition (ICPR), Beijing, China.","DOI":"10.1109\/ICPR.2018.8546006"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"10227","DOI":"10.1109\/TGRS.2020.3042974","article-title":"PSGAN: A generative adversarial network for remote sensing image pan-sharpening","volume":"59","author":"Liu","year":"2020","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Ram Prabhakar, K., Sai Srikar, V., and Venkatesh Babu, R. (2017, January 22\u201329). Deepfuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.505"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"110","DOI":"10.1016\/j.inffus.2020.04.006","article-title":"Pan-GAN: An unsupervised pan-sharpening method for remote sensing image fusion","volume":"62","author":"Ma","year":"2020","journal-title":"Inf. Fusion"},{"key":"ref_42","unstructured":"Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. Adv. Neural Inf. Process. Syst., 27, Available online: https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2014\/file\/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf."},{"key":"ref_43","unstructured":"Mirza, M., and Osindero, S. (2014). Conditional generative adversarial nets. arXiv."},{"key":"ref_44","unstructured":"Durugkar, I., Gemp, I., and Mahadevan, S. (2016). Generative Multi-Adversarial Networks. arXiv."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Wang, L., Sindagi, V., and Patel, V. (2018, January 15\u201319). High-quality facial photo-sketch synthesis using multi-adversarial networks. Proceedings of the 13th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2018), Xi\u2019an, China.","DOI":"10.1109\/FG.2018.00022"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Aghakhani, H., Machiry, A., Nilizadeh, S., Kruegel, C., and Vigna, G. (2018, January 24\u201324). Detecting deceptive reviews using generative adversarial networks. Proceedings of the 2018 IEEE Security and Privacy Workshops (SPW), San Francisco, CA, USA.","DOI":"10.1109\/SPW.2018.00022"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"971","DOI":"10.1109\/TPAMI.2002.1017623","article-title":"Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns","volume":"24","author":"Ojala","year":"2002","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"915","DOI":"10.1109\/TPAMI.2007.1110","article-title":"Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions","volume":"29","author":"Zhao","year":"2007","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Maturana, D., Mery, D., and Soto, \u00c1. (2009, January 10\u201312). Face Recognition with Local Binary Patterns, Spatial Pyramid Histograms and Naive Bayes Nearest Neighbor Classification. Proceedings of the 2009 International Conference of the Chilean Computer Science Society, Santiago, Chile.","DOI":"10.1109\/SCCC.2009.21"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Tapia, J.E., Perez, C.A., and Bowyer, K.W. (12, January 6\u20137). Gender Classification from Iris Images Using Fusion of Uniform Local Binary Patterns. Proceedings of the Computer Vision\u2014ECCV 2014 Workshops, Zurich, Switzerland.","DOI":"10.1007\/978-3-319-16181-5_57"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21\u201326). Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.243"},{"key":"ref_52","first-page":"1097","article-title":"Imagenet classification with deep convolutional neural networks","volume":"25","author":"Krizhevsky","year":"2012","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"600","DOI":"10.1109\/TIP.2003.819861","article-title":"Image quality assessment: From error visibility to structural similarity","volume":"13","author":"Wang","year":"2004","journal-title":"IEEE Trans. Image Process."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1016\/j.inffus.2021.02.008","article-title":"An infrared and visible image fusion method based on multi-scale transformation and norm optimization","volume":"71","author":"Li","year":"2021","journal-title":"Inf. Fusion"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"102820","DOI":"10.1016\/j.trc.2020.102820","article-title":"Risk assessment based collision avoidance decision-making for autonomous vehicles in multi-scenarios","volume":"122","author":"Li","year":"2021","journal-title":"Transp. Res. Part Emerg. Technol."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1016\/j.trc.2016.11.011","article-title":"Estimation of driving style in naturalistic highway traffic using maneuver transition probabilities","volume":"74","author":"Li","year":"2017","journal-title":"Transp. Res. Part Emerg. Technol."},{"key":"ref_57","unstructured":"(2023, March 10). AMPS Programme. September 1998, Available online: http:\/\/info.amps.gov:2080."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"2959","DOI":"10.1109\/26.477498","article-title":"Image quality measures and their performance","volume":"43","author":"Eskicioglu","year":"1995","journal-title":"IEEE Trans. Commun."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1016\/j.optcom.2014.12.032","article-title":"Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scale image decomposition","volume":"341","author":"Cui","year":"2015","journal-title":"Opt. Commun."},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"313","DOI":"10.1049\/el:20020212","article-title":"Information measure for performance of image fusion","volume":"38","author":"Qu","year":"2002","journal-title":"Electron. Lett."},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"023522","DOI":"10.1117\/1.2945910","article-title":"Assessment of image fusion procedures using entropy, image quality, and multispectral classification","volume":"2","author":"Roberts","year":"2008","journal-title":"J. Appl. Remote Sens."},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1109\/97.995823","article-title":"A universal image quality index","volume":"9","author":"Wang","year":"2002","journal-title":"IEEE Signal Process. Lett."},{"key":"ref_63","doi-asserted-by":"crossref","first-page":"127","DOI":"10.1016\/j.inffus.2011.08.002","article-title":"A new image fusion performance metric based on visual information fidelity","volume":"14","author":"Han","year":"2013","journal-title":"Inf. Fusion"},{"key":"ref_64","doi-asserted-by":"crossref","unstructured":"Du, Q., Xu, H., Ma, Y., Huang, J., and Fan, F. (2018). Fusing infrared and visible images of different resolutions via total variation model. Sensors, 18.","DOI":"10.3390\/s18113827"},{"key":"ref_65","doi-asserted-by":"crossref","first-page":"5258","DOI":"10.1109\/TGRS.2020.3014698","article-title":"Fusionndvi: A computational fusion approach for high-resolution normalized difference vegetation index","volume":"59","author":"Tian","year":"2020","journal-title":"IEEE Trans. Geosci. Remote Sens."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/15\/9\/2440\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:30:21Z","timestamp":1760124621000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/15\/9\/2440"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,6]]},"references-count":65,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2023,5]]}},"alternative-id":["rs15092440"],"URL":"https:\/\/doi.org\/10.3390\/rs15092440","relation":{},"ISSN":["2072-4292"],"issn-type":[{"type":"electronic","value":"2072-4292"}],"subject":[],"published":{"date-parts":[[2023,5,6]]}}}