{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,16]],"date-time":"2026-03-16T05:09:11Z","timestamp":1773637751344,"version":"3.50.1"},"reference-count":51,"publisher":"MDPI AG","issue":"21","license":[{"start":{"date-parts":[[2024,10,26]],"date-time":"2024-10-26T00:00:00Z","timestamp":1729900800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Key Research and Development Program of China","award":["2021ZD0112902"],"award-info":[{"award-number":["2021ZD0112902"]}]},{"name":"National Key Research and Development Program of China","award":["62272375"],"award-info":[{"award-number":["62272375"]}]},{"name":"National Key Research and Development Program of China","award":["12226004"],"award-info":[{"award-number":["12226004"]}]},{"name":"China NSFC Projects","award":["2021ZD0112902"],"award-info":[{"award-number":["2021ZD0112902"]}]},{"name":"China NSFC Projects","award":["62272375"],"award-info":[{"award-number":["62272375"]}]},{"name":"China NSFC Projects","award":["12226004"],"award-info":[{"award-number":["12226004"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Multispectral and hyperspectral image fusion (MS\/HS fusion) aims to generate a high-resolution hyperspectral (HRHS) image by fusing a high-resolution multispectral (HRMS) and a low-resolution hyperspectral (LRHS) images. The deep unfolding-based MS\/HS fusion method is a representative deep learning paradigm due to its excellent performance and sufficient interpretability. However, existing deep unfolding-based MS\/HS fusion methods only rely on a fixed linear degradation model, which focuses on modeling the relationships between HRHS and HRMS, as well as HRHS and LRHS. In this paper, we break free from this observation model framework and propose a new observation model. Firstly, the proposed observation model is built based on the convolutional sparse coding (CSC) technique, and then a proximal gradient algorithm is designed to solve this model. Secondly, we unfold the iterative algorithm into a deep network, dubbed as MHF-CSCNet, where the proximal operators are learned using convolutional neural networks. Finally, all trainable parameters can be automatically learned end-to-end from the training pairs. Experimental evaluations conducted on various benchmark datasets demonstrate the superiority of our method both quantitatively and qualitatively compared to other state-of-the-art methods.<\/jats:p>","DOI":"10.3390\/rs16213979","type":"journal-article","created":{"date-parts":[[2024,10,28]],"date-time":"2024-10-28T07:04:04Z","timestamp":1730099044000},"page":"3979","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["A Deep Unfolding Network for Multispectral and Hyperspectral Image Fusion"],"prefix":"10.3390","volume":"16","author":[{"given":"Bihui","family":"Zhang","sequence":"first","affiliation":[{"name":"School of Mathematics and Statistics, Xi\u2019an Jiaotong University, Xi\u2019an 710049, China"},{"name":"The Ministry of Education Key Lab for Intelligent Networks and Network Security, Xi\u2019an Jiaotong University, Xi\u2019an 710049, China"}]},{"given":"Xiangyong","family":"Cao","sequence":"additional","affiliation":[{"name":"The Ministry of Education Key Lab for Intelligent Networks and Network Security, Xi\u2019an Jiaotong University, Xi\u2019an 710049, China"},{"name":"School of Computer Science and Technology, Xi\u2019an Jiaotong University, Xi\u2019an 710049, China"}]},{"given":"Deyu","family":"Meng","sequence":"additional","affiliation":[{"name":"School of Mathematics and Statistics, Xi\u2019an Jiaotong University, Xi\u2019an 710049, China"},{"name":"The Peng Cheng Laboratory, Shenzhen 518066, China"},{"name":"The Macau Institute of Systems Engineering, Macau University of Science and Technology, Taipa, Macau 999078, China"}]}],"member":"1968","published-online":{"date-parts":[[2024,10,26]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1109\/MGRS.2016.2637824","article-title":"Hyperspectral and Multispectral Data Fusion: A comparative review of the recent literature","volume":"5","author":"Yokoya","year":"2017","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"116","DOI":"10.1109\/TIP.2019.2928895","article-title":"Super-Resolution for Hyperspectral and Multispectral Image Fusion Accounting for Seasonal Spectral Variability","volume":"29","author":"Borsoi","year":"2020","journal-title":"IEEE Trans. Image Process."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"5527812","DOI":"10.1109\/TGRS.2023.3324497","article-title":"Decoupled-and-Coupled Networks: Self-Supervised Hyperspectral Image Super-Resolution with Subpixel Fusion","volume":"61","author":"Hong","year":"2023","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Liu, N., Li, W., and Tao, R. (2022, January 7\u201313). Geometric Low-Rank Tensor Approximation for Remotely Sensed Hyperspectral And Multispectral Imagery Fusion. Proceedings of the ICASSP 2022\u20142022 IEEE International Conference On Acoustics, Speech and Signal Processing (ICASSP), Virtual.","DOI":"10.1109\/ICASSP43922.2022.9746041"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"7654","DOI":"10.1109\/TGRS.2020.2983063","article-title":"Hyperspectral and Multispectral Image Fusion via Nonlocal Low-Rank Tensor Decomposition and Spectral Unmixing","volume":"58","author":"Wang","year":"2020","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"8845","DOI":"10.1109\/JSTARS.2023.3314085","article-title":"FS-Net: Four-Stream Network With Spatial\u2013Spectral Representation Learning for Hyperspectral and Multispecral Image Fusion","volume":"16","author":"Jin","year":"2023","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"3230","DOI":"10.1109\/TGRS.2007.901007","article-title":"Improving component substitution pansharpening through multivariate regression of MS + Pan data","volume":"45","author":"Aiazzi","year":"2007","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"3461","DOI":"10.1080\/014311600750037499","article-title":"Smoothing filter-based intensity modulation: A spectral preserve image fusion technique for improving spatial details","volume":"21","author":"Liu","year":"2000","journal-title":"Int. J. Remote Sens."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"591","DOI":"10.14358\/PERS.72.5.591","article-title":"MTF-tailored multiscale fusion of high-resolution MS and Pan imagery","volume":"72","author":"Aiazzi","year":"2006","journal-title":"Photogramm. Eng. Remote Sens."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"3373","DOI":"10.1109\/TGRS.2014.2375320","article-title":"A convex formulation for hyperspectral image superresolution via subspace-based regularization","volume":"53","author":"Simoes","year":"2014","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"528","DOI":"10.1109\/TGRS.2011.2161320","article-title":"Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion","volume":"50","author":"Yokoya","year":"2011","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Akhtar, N., Shafait, F., and Mian, A. (2014, January 6\u201312). Sparse spatio-spectral representation for hyperspectral image super-resolution. Proceedings of the Computer Vision\u2014ECCV 2014: 13th European Conference, Zurich, Switzerland. Proceedings, Part VII 13.","DOI":"10.1007\/978-3-319-10584-0_5"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Lanaras, C., Baltsavias, E., and Schindler, K. (2015, January 7\u201313). Hyperspectral super-resolution by coupled spectral unmixing. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.409"},{"key":"ref_14","unstructured":"Eismann, M. (2004). Resolution Enhancement of Hyperspectral Imagery Using Maximum a Posteriori Estimation with a Stochastic Mixing Model. [Ph.D. Thesis, University of Dayton]."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"4109","DOI":"10.1109\/TIP.2015.2458572","article-title":"Fast fusion of multi-band images based on solving a Sylvester equation","volume":"24","author":"Wei","year":"2015","journal-title":"IEEE Trans. Image Process."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"4118","DOI":"10.1109\/TIP.2018.2836307","article-title":"Fusing hyperspectral and multispectral images via coupled sparse tensor factorization","volume":"27","author":"Li","year":"2018","journal-title":"IEEE Trans. Image Process."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"4381","DOI":"10.1109\/JSTARS.2020.3012566","article-title":"Hyperspectral image superresolution using unidirectional total variation with tucker decomposition","volume":"13","author":"Xu","year":"2020","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_18","unstructured":"Laben, C., and Brower, B. (2000). Process for Enhancing the Spatial Resolution of Multispectral Imagery Using Pan-Sharpening. (6,011,875), U.S. Patent."},{"key":"ref_19","first-page":"459","article-title":"Others The use of intensity-hue-saturation transformations for merging SPOT panchromatic and multispectral image data","volume":"56","author":"Carper","year":"1990","journal-title":"Photogramm. Eng. Remote Sens."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"3008","DOI":"10.1109\/JSTARS.2015.2440092","article-title":"Hyper-sharpening: A first approach on SIM-GA data","volume":"8","author":"Selva","year":"2015","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"898","DOI":"10.1109\/TGRS.2005.844293","article-title":"Vertex component analysis: A fast algorithm to unmix hyperspectral data","volume":"43","author":"Nascimento","year":"2005","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"4349","DOI":"10.1109\/TGRS.2018.2890705","article-title":"CoSpace: Common Subspace Learning From Hyperspectral-Multispectral Correspondences","volume":"57","author":"Hong","year":"2019","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"788","DOI":"10.1038\/44565","article-title":"Learning the parts of objects by non-negative matrix factorization","volume":"401","author":"Lee","year":"1999","journal-title":"Nature"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Yang, J., Zhao, Y., and Chan, J. (2018). Hyperspectral and multispectral image fusion via deep two-branches convolutional neural network. Remote Sens., 10.","DOI":"10.3390\/rs10050800"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1016\/j.neucom.2017.05.024","article-title":"Hyperspectral image super-resolution using deep convolutional neural network","volume":"266","author":"Li","year":"2017","journal-title":"Neurocomputing"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"11276","DOI":"10.1109\/TIE.2020.3038096","article-title":"Hyperspectral image superresolution using spectrum and feature context","volume":"68","author":"Wang","year":"2020","journal-title":"IEEE Trans. Ind. Electron."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1233","DOI":"10.1109\/TCI.2020.3014451","article-title":"Deep recursive network for hyperspectral image super-resolution","volume":"6","author":"Wei","year":"2020","journal-title":"IEEE Trans. Comput. Imaging"},{"key":"ref_28","first-page":"6012305","article-title":"Fusformer: A transformer-based fusion network for hyperspectral image super-resolution","volume":"19","author":"Hu","year":"2022","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Han, X., Yu, J., Luo, J., and Sun, W. (2019). Hyperspectral and multispectral image fusion using cluster-based multi-branch BP neural networks. Remote Sens., 11.","DOI":"10.3390\/rs11101173"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1124","DOI":"10.1109\/TNNLS.2020.2980398","article-title":"Regularizing hyperspectral and multispectral image fusion by CNN denoiser","volume":"32","author":"Dian","year":"2020","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"7251","DOI":"10.1109\/TNNLS.2021.3084682","article-title":"Hyperspectral image super-resolution via deep spatiospectral attention convolutional neural networks","volume":"33","author":"Hu","year":"2021","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Palsson, F., Sveinsson, J., and Ulfarsson, M. (2018). Sentinel-2 image fusion using a deep residual network. Remote Sens., 10.","DOI":"10.3390\/rs10081290"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"5635318","DOI":"10.1109\/TGRS.2022.3229027","article-title":"Unsupervised Multimodal Change Detection Based on Structural Relationship Graph Representation Learning","volume":"60","author":"Chen","year":"2022","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1016\/j.isprsjprs.2023.03.004","article-title":"Fourier domain structural relationship analysis for unsupervised multimodal change detection","volume":"198","author":"Chen","year":"2023","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Xie, Q., Zhou, M., Zhao, Q., Meng, D., Zuo, W., and Xu, Z. (2019, January 15\u201320). Multispectral and hyperspectral image fusion by MS\/HS fusion net. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00168"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"5754","DOI":"10.1109\/TIP.2021.3078058","article-title":"Model-guided deep hyperspectral image super-resolution","volume":"30","author":"Dong","year":"2021","journal-title":"IEEE Trans. Image Process."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1109\/TCI.2021.3136759","article-title":"Deep Unfolding Network for Spatiospectral Image Super-Resolution","volume":"8","author":"Ma","year":"2022","journal-title":"IEEE Trans. Comput. Imaging"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1250","DOI":"10.1080\/2150704X.2021.1979270","article-title":"A deep image prior-based interpretable network for hyperspectral image fusion","volume":"12","author":"Sun","year":"2021","journal-title":"Remote Sens. Lett."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"3333","DOI":"10.1109\/TPAMI.2020.2984244","article-title":"Deep convolutional neural network for multi-modal image restoration and fusion","volume":"43","author":"Deng","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"2029","DOI":"10.1109\/TIP.2021.3050313","article-title":"Online rain\/snow removal from surveillance videos","volume":"30","author":"Li","year":"2021","journal-title":"IEEE Trans. Image Process."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"166","DOI":"10.1109\/JSTARS.2022.3220974","article-title":"Panchromatic and Hyperspectral Image Fusion: Outcome of the 2022 WHISPERS Hyperspectral Pansharpening Challenge","volume":"16","author":"Vivone","year":"2023","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"2241","DOI":"10.1109\/TIP.2010.2046811","article-title":"Generalized assorted pixel camera: Postcapture control of resolution, dynamic range, and spectrum","volume":"19","author":"Yasuma","year":"2010","journal-title":"IEEE Trans. Image Process."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Chakrabarti, A., and Zickler, T. (2011, January 20\u201325). Statistics of real-world hyperspectral images. Proceedings of the CVPR 2011, Colorado Springs, CO, USA.","DOI":"10.1109\/CVPR.2011.5995660"},{"key":"ref_44","unstructured":"Yokoya, N., and Iwasaki, A. (2016). Airborne Hyperspectral Data over Chikusei, Space Application Laboratory, the University of Tokyo. Technical Report SAL-2016-05-27."},{"key":"ref_45","unstructured":"Yuhas, R., Boardman, J., and Goetz, A. (1993). Determination of semi-arid landscape endmembers and seasonal trends using convex geometry spectral unmixing techniques. Summaries of the 4th Annual JPL Airborne Geoscience Workshop. Volume 1: AVIRIS Workshop, JPL."},{"key":"ref_46","unstructured":"Wald, L. (2000, January 26\u201328). Quality of high resolution synthesised images: Is there a simple criterion?. Proceedings of the Third Conference \u201cFusion of Earth Data: Merging Point Measurements, Raster Maps and Remotely Sensed Images\u201d, Sophia Antipolis, France."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"600","DOI":"10.1109\/TIP.2003.819861","article-title":"Image quality assessment: From error visibility to structural similarity","volume":"13","author":"Wang","year":"2004","journal-title":"IEEE Trans. Image Process."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"662","DOI":"10.1109\/LGRS.2009.2022650","article-title":"Hypercomplex quality assessment of multi\/hyperspectral images","volume":"6","author":"Garzelli","year":"2009","journal-title":"IEEE Geosci. Remote. Sens. Lett."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1109\/97.995823","article-title":"A universal image quality index","volume":"9","author":"Wang","year":"2002","journal-title":"IEEE Signal Process. Lett."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"2565","DOI":"10.1109\/TGRS.2014.2361734","article-title":"A Critical Comparison Among Pansharpening Algorithms","volume":"53","author":"Vivone","year":"2015","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"193","DOI":"10.14358\/PERS.74.2.193","article-title":"Multispectral and panchromatic data fusion assessment without reference","volume":"74","author":"Alparone","year":"2008","journal-title":"Photogramm. Eng. Remote Sens."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/16\/21\/3979\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T16:21:11Z","timestamp":1760113271000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/16\/21\/3979"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10,26]]},"references-count":51,"journal-issue":{"issue":"21","published-online":{"date-parts":[[2024,11]]}},"alternative-id":["rs16213979"],"URL":"https:\/\/doi.org\/10.3390\/rs16213979","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,10,26]]}}}