{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,22]],"date-time":"2026-02-22T10:00:44Z","timestamp":1771754444197,"version":"3.50.1"},"reference-count":36,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2023,1,14]],"date-time":"2023-01-14T00:00:00Z","timestamp":1673654400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Multimodal image fusion aims to retain valid information from different modalities, remove redundant information to highlight critical targets, and maintain rich texture details in the fused image. However, current image fusion networks only use simple convolutional layers to extract features, ignoring global dependencies and channel contexts. This paper proposes GRPAFusion, a multimodal image fusion framework based on gradient residual and pyramid attention. The framework uses multiscale gradient residual blocks to extract multiscale structural features and multigranularity detail features from the source image. The depth features from different modalities were adaptively corrected for inter-channel responses using a pyramid split attention module to generate high-quality fused images. Experimental results on public datasets indicated that GRPAFusion outperforms the current fusion methods in subjective and objective evaluations.<\/jats:p>","DOI":"10.3390\/e25010169","type":"journal-article","created":{"date-parts":[[2023,1,16]],"date-time":"2023-01-16T02:29:55Z","timestamp":1673836195000},"page":"169","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["GRPAFusion: A Gradient Residual and Pyramid Attention-Based Multiscale Network for Multimodal Image Fusion"],"prefix":"10.3390","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4205-7673","authenticated-orcid":false,"given":"Jinxin","family":"Wang","sequence":"first","affiliation":[{"name":"Optoelectronic System Laboratory, Institute of Semiconductors, Chinese Academy of Sciences, Beijing 100083, China"},{"name":"College of Materials Science and Opto-Electronic Technology, University of Chinese Academy of Sciences, Beijing 100049, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7242-5695","authenticated-orcid":false,"given":"Xiaoli","family":"Xi","sequence":"additional","affiliation":[{"name":"Optoelectronic System Laboratory, Institute of Semiconductors, Chinese Academy of Sciences, Beijing 100083, China"},{"name":"College of Materials Science and Opto-Electronic Technology, University of Chinese Academy of Sciences, Beijing 100049, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6655-8419","authenticated-orcid":false,"given":"Dongmei","family":"Li","sequence":"additional","affiliation":[{"name":"Optoelectronic System Laboratory, Institute of Semiconductors, Chinese Academy of Sciences, Beijing 100083, China"},{"name":"College of Materials Science and Opto-Electronic Technology, University of Chinese Academy of Sciences, Beijing 100049, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9420-1097","authenticated-orcid":false,"given":"Fang","family":"Li","sequence":"additional","affiliation":[{"name":"Optoelectronic System Laboratory, Institute of Semiconductors, Chinese Academy of Sciences, Beijing 100083, China"},{"name":"College of Materials Science and Opto-Electronic Technology, University of Chinese Academy of Sciences, Beijing 100049, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1403-4036","authenticated-orcid":false,"given":"Guanxin","family":"Zhang","sequence":"additional","affiliation":[{"name":"Optoelectronic System Laboratory, Institute of Semiconductors, Chinese Academy of Sciences, Beijing 100083, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,1,14]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1771","DOI":"10.1016\/j.patcog.2006.11.010","article-title":"Fusion of Color and Infrared Video for Moving Human Detection","volume":"40","author":"Han","year":"2007","journal-title":"Pattern Recognit."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"323","DOI":"10.1016\/j.inffus.2021.06.008","article-title":"Image Fusion Meets Deep Learning: A Survey and Perspective","volume":"76","author":"Zhang","year":"2021","journal-title":"Inf. Fusion"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"142101","DOI":"10.1007\/s11432-020-3160-5","article-title":"RGBT Tracking via Reliable Feature Configuration","volume":"65","author":"Tu","year":"2022","journal-title":"Sci. China Inf. Sci."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1016\/j.inffus.2021.12.004","article-title":"Image Fusion in the Loop of High-Level Vision Tasks: A Semantic-Aware Real-Time Infrared and Visible Image Fusion Network","volume":"82","author":"Tang","year":"2022","journal-title":"Inf. Fusion"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Ha, Q., Watanabe, K., Karasawa, T., Ushiku, Y., and Harada, T. (2017, January 24\u201328). MFNet: Towards Real-Time Semantic Segmentation for Autonomous Vehicles with Multi-Spectral Scenes. Proceedings of the 2017 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada.","DOI":"10.1109\/IROS.2017.8206396"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1109\/JSEN.2015.2478655","article-title":"Fusion of infrared and visible sensor images based on anisotropic diffusion and Karhunen-Loeve transform","volume":"16","author":"Bavirisetti","year":"2015","journal-title":"IEEE Sensors J."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"479","DOI":"10.14429\/dsj.61.705","article-title":"Image fusion technique using multi-resolution singular value decomposition","volume":"61","author":"Naidu","year":"2011","journal-title":"Def. Sci. J."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1193","DOI":"10.1007\/s11760-013-0556-9","article-title":"Image fusion based on pixel significance using cross bilateral filter","volume":"9","year":"2015","journal-title":"Signal Image Video Process."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"6480","DOI":"10.1364\/AO.55.006480","article-title":"Fusion of infrared and visible images for night-vision context enhancement","volume":"55","author":"Zhou","year":"2016","journal-title":"Appl. Opt."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"2864","DOI":"10.1109\/TIP.2013.2244222","article-title":"Image fusion with guided filtering","volume":"22","author":"Li","year":"2013","journal-title":"IEEE Trans. Image Process."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1016\/j.inffus.2014.09.004","article-title":"A General Framework for Image Fusion Based on Multi-Scale Transform and Sparse Representation","volume":"24","author":"Z","year":"2015","journal-title":"Inf. Fusion"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1882","DOI":"10.1109\/LSP.2016.2618776","article-title":"Image Fusion With Convolutional Sparse Representation","volume":"23","author":"Liu","year":"2016","journal-title":"IEEE Signal Process. Lett."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"94","DOI":"10.1016\/j.infrared.2017.04.018","article-title":"Infrared and Visible Image Fusion Method Based on Saliency Detection in Sparse Domain","volume":"83","author":"Liu","year":"2017","journal-title":"Infrared Phys. Technol."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1016\/j.infrared.2016.01.009","article-title":"Two-scale image fusion of visible and infrared images using saliency detection","volume":"76","author":"Bavirisetti","year":"2016","journal-title":"Infrared Phys. Technol."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1016\/j.infrared.2017.02.005","article-title":"Infrared and visible image fusion based on visual saliency map and weighted least square optimization","volume":"82","author":"Ma","year":"2017","journal-title":"Infrared Phys. Technol."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1016\/j.neucom.2012.12.015","article-title":"Fast saliency-aware multi-modality image fusion","volume":"111","author":"Han","year":"2013","journal-title":"Neurocomputing"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1850018","DOI":"10.1142\/S0219691318500182","article-title":"Infrared and Visible Image Fusion with Convolutional Neural Networks","volume":"16","author":"Liu","year":"2018","journal-title":"Int. J. Wavelets Multiresolution Inf. Process."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Li, H., Wu, X.J., and Kittler, J. (2018, January 20\u201324). Infrared and Visible Image Fusion Using a Deep Learning Framework. Proceedings of the 2018 24th International Conference on Pattern Recognition (ICPR), Beijing, China.","DOI":"10.1109\/ICPR.2018.8546006"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1016\/j.inffus.2018.09.004","article-title":"FusionGAN: A Generative Adversarial Network for Infrared and Visible Image Fusion","volume":"48","author":"Ma","year":"2019","journal-title":"Inf. Fusion"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"2614","DOI":"10.1109\/TIP.2018.2887342","article-title":"DenseFuse: A Fusion Approach to Infrared and Visible Images","volume":"28","author":"Li","year":"2019","journal-title":"IEEE Trans. Image Process."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"104486","DOI":"10.1016\/j.infrared.2022.104486","article-title":"Infrared and Visible Image Fusion Based on Residual Dense Network and Gradient Loss","volume":"128","author":"Li","year":"2023","journal-title":"Infrared Phys. Technol."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1383","DOI":"10.1109\/TMM.2020.2997127","article-title":"AttentionFGAN: Infrared and Visible Image Fusion Using Attention-Based Generative Adversarial Networks","volume":"23","author":"Li","year":"2021","journal-title":"IEEE Trans. Multimed."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1200","DOI":"10.1109\/JAS.2022.105686","article-title":"SwinFusion: Cross-Domain Long-Range Learning for General Image Fusion via Swin Transformer","volume":"9","author":"Ma","year":"2022","journal-title":"IEEE\/CAA J. Autom. Sin."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"2121","DOI":"10.1109\/JAS.2022.106082","article-title":"SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness","volume":"9","author":"Tang","year":"2022","journal-title":"IEEE\/CAA J. Autom. Sin."},{"key":"ref_25","unstructured":"Zhang, H., Zu, K., Lu, J., Zou, Y., and Meng, D. (2021). EPSANet: An efficient pyramid squeeze attention block on convolutional neural network. arXiv."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"652","DOI":"10.1109\/TPAMI.2019.2938758","article-title":"Res2Net: A New Multi-Scale Backbone Architecture","volume":"43","author":"Gao","year":"2021","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Hu, J., Shen, L., and Sun, G. (2018, January 18\u201322). Squeeze-and-excitation networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00745"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"023522","DOI":"10.1117\/1.2945910","article-title":"Assessment of image fusion procedures using entropy, image quality, and multispectral classification","volume":"2","author":"Roberts","year":"2008","journal-title":"J. Appl. Remote Sens."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1016\/j.optcom.2014.12.032","article-title":"Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scale image decomposition","volume":"341","author":"Cui","year":"2015","journal-title":"Opt. Commun."},{"key":"ref_30","unstructured":"Piella, G., and Heijmans, H. (2003, January 14\u201317). A New Quality Metric for Image Fusion. Proceedings of the 2003 International Conference on Image Processing (Cat. No.03CH37429), Barcelona, Spain."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Haghighat, M., and Razian, M.A. (2014, January 15\u201317). Fast-FMI: Non-reference image fusion metric. Proceedings of the 2014 IEEE 8th International Conference on Application of Information and Communication Technologies (AICT), Astana, Kazakhstan.","DOI":"10.1109\/ICAICT.2014.7036000"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"3345","DOI":"10.1109\/TIP.2015.2442920","article-title":"Perceptual Quality Assessment for Multi-Exposure Image Fusion","volume":"24","author":"Ma","year":"2015","journal-title":"IEEE Trans. Image Process."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1016\/j.dib.2017.09.038","article-title":"The TNO Multiband Image Data Collection","volume":"15","author":"Toet","year":"2017","journal-title":"Data Brief"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Jia, X., Zhu, C., Li, M., Tang, W., and Zhou, W. (2021, January 19\u201325). LLVIP: A visible-infrared paired dataset for low-light vision. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Montreal, QC, Canada.","DOI":"10.1109\/ICCVW54120.2021.00389"},{"key":"ref_35","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"502","DOI":"10.1109\/TPAMI.2020.3012548","article-title":"U2Fusion: A Unified Unsupervised Image Fusion Network","volume":"44","author":"Xu","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/25\/1\/169\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:05:58Z","timestamp":1760119558000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/25\/1\/169"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,14]]},"references-count":36,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,1]]}},"alternative-id":["e25010169"],"URL":"https:\/\/doi.org\/10.3390\/e25010169","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,14]]}}}