{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,15]],"date-time":"2026-04-15T18:26:42Z","timestamp":1776277602318,"version":"3.50.1"},"reference-count":41,"publisher":"MDPI AG","issue":"16","license":[{"start":{"date-parts":[[2021,8,13]],"date-time":"2021-08-13T00:00:00Z","timestamp":1628812800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62071341"],"award-info":[{"award-number":["62071341"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Compared with multispectral sensors, hyperspectral sensors obtain images with high- spectral resolution at the cost of spatial resolution, which constrains the further and precise application of hyperspectral images. An intelligent idea to obtain high-resolution hyperspectral images is hyperspectral and multispectral image fusion. In recent years, many studies have found that deep learning-based fusion methods outperform the traditional fusion methods due to the strong non-linear fitting ability of convolution neural network. However, the function of deep learning-based methods heavily depends on the size and quality of training dataset, constraining the application of deep learning under the situation where training dataset is not available or of low quality. In this paper, we introduce a novel fusion method, which operates in a self-supervised manner, to the task of hyperspectral and multispectral image fusion without training datasets. Our method proposes two constraints constructed by low-resolution hyperspectral images and fake high-resolution hyperspectral images obtained from a simple diffusion method. Several simulation and real-data experiments are conducted with several popular remote sensing hyperspectral data under the condition where training datasets are unavailable. Quantitative and qualitative results indicate that the proposed method outperforms those traditional methods by a large extent.<\/jats:p>","DOI":"10.3390\/rs13163226","type":"journal-article","created":{"date-parts":[[2021,8,15]],"date-time":"2021-08-15T22:51:27Z","timestamp":1629067887000},"page":"3226","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["Hyperspectral and Multispectral Image Fusion by Deep Neural Network in a Self-Supervised Manner"],"prefix":"10.3390","volume":"13","author":[{"given":"Jianhao","family":"Gao","sequence":"first","affiliation":[{"name":"School of Geodesy and Geomatics, Wuhan University, Wuhan 430072, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4063-9381","authenticated-orcid":false,"given":"Jie","family":"Li","sequence":"additional","affiliation":[{"name":"School of Geodesy and Geomatics, Wuhan University, Wuhan 430072, China"}]},{"given":"Menghui","family":"Jiang","sequence":"additional","affiliation":[{"name":"School of Resource and Environmental Sciences, Wuhan University, Wuhan 430072, China"}]}],"member":"1968","published-online":{"date-parts":[[2021,8,13]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1793","DOI":"10.1109\/TGRS.2015.2488681","article-title":"Scene classification via a gradient boosting random convolutional network framework","volume":"54","author":"Zhang","year":"2015","journal-title":"IEEE Trans. Geosci. Remote. Sens."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Uzkent, B., Rangnekar, A., and Hoffman, M. (2017, January 21\u201326). Aerial vehicle tracking by adaptive fusion of hyperspectral likelihood maps. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA.","DOI":"10.1109\/CVPRW.2017.35"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"4103","DOI":"10.1109\/TGRS.2011.2167193","article-title":"Foreword to the special issue on spectral unmixing of remotely sensed data","volume":"49","author":"Plaza","year":"2011","journal-title":"IEEE Trans. Geosci. Remote. Sens."},{"key":"ref_4","first-page":"245","article-title":"A comprehensive drought monitoring method integrating MODIS and TRMM data","volume":"23","author":"Du","year":"2013","journal-title":"Int. J. Appl. Earth Obs. Geoinf."},{"key":"ref_5","first-page":"53","article-title":"Forest biomass retrieval approaches from earth observation in different biomes","volume":"77","author":"Quegan","year":"2019","journal-title":"Int. J. Appl. Earth Obs. Geoinf."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1109\/MGRS.2016.2637824","article-title":"Hyperspectral and multispectral data fusion: A comparative review of the recent literature","volume":"5","author":"Yokoya","year":"2017","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_7","first-page":"339","article-title":"Extracting spectral contrast in Landsat Thematic Mapper image data using selective principal component analysis","volume":"55","author":"Kwarteng","year":"1989","journal-title":"Photogramm. Eng. Remote. Sens."},{"key":"ref_8","first-page":"459","article-title":"The use of intensity-hue-saturation transformations for merging SPOT panchromatic and multispectral image data","volume":"56","author":"Carper","year":"1990","journal-title":"Photogramm. Eng. Remote. Sens."},{"key":"ref_9","unstructured":"Laben, C.A., and Brower, B.V. (2000). Process for Enhancing the Spatial Resolution of Multispectral Imagery Using Pan-Sharpening. (No. 6,011,875), U.S. Patent."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"3230","DOI":"10.1109\/TGRS.2007.901007","article-title":"Improving component substitution pansharpening through multivariate regression of MS + Pan data","volume":"45","author":"Aiazzi","year":"2007","journal-title":"IEEE Trans. Geosci. Remote. Sens."},{"key":"ref_11","first-page":"295","article-title":"Comparison of three different methods to merge multiresolution and multispectral data- Landsat TM and SPOT panchromatic","volume":"57","author":"Chavez","year":"1991","journal-title":"Photogramm. Eng. Remote. Sens."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"3461","DOI":"10.1080\/014311600750037499","article-title":"Smoothing filter-based intensity modulation: A spectral preserve image fusion technique for improving spatial details","volume":"21","author":"Liu","year":"2000","journal-title":"Int. J. Remote. Sens."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"591","DOI":"10.14358\/PERS.72.5.591","article-title":"MTF-tailored multiscale fusion of high-resolution MS and Pan imagery","volume":"72","author":"Aiazzi","year":"2006","journal-title":"Photogramm. Eng. Remote. Sens."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"826","DOI":"10.1109\/LGRS.2017.2682122","article-title":"Pansharpening of clustered MS and Pan images considering mixed pixels","volume":"14","author":"Shahdoosti","year":"2017","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"2241","DOI":"10.1109\/TIP.2010.2046811","article-title":"Generalized assorted pixel camera: Postcapture control of resolution, dynamic range, and spectrum","volume":"19","author":"Yasuma","year":"2010","journal-title":"IEEE Trans. Image Process."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"2014","DOI":"10.1109\/TGRS.2010.2098413","article-title":"Sparse unmixing of hyperspectral data","volume":"49","author":"Iordache","year":"2011","journal-title":"IEEE Trans. Geosci. Remote. Sens."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"528","DOI":"10.1109\/TGRS.2011.2161320","article-title":"Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion","volume":"50","author":"Yokoya","year":"2011","journal-title":"IEEE Trans. Geosci. Remote. Sens."},{"key":"ref_18","unstructured":"Akhtar, N., Shafait, F., and Mian, A. (2020, January 23\u201328). Sparse spatio-spectral representation for hyperspectral image super-resolution. Proceedings of the European Conference on Computer Vision, Glasgow, UK."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Akhtar, N., Shafait, F., and Mian, A. (2015, January 7\u201312). Bayesian sparse representation for hyperspectral image super resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298986"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Lanaras, C., Baltsavias, E., and Schindler, K. (2015, January 7\u201313). Hyperspectral super-resolution by coupled spectral unmixing. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.409"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1693","DOI":"10.1109\/TSMCB.2012.2198810","article-title":"Adjustable model-based fusion method for multispectral and panchromatic images","volume":"42","author":"Zhang","year":"2012","journal-title":"IEEE Trans. Syst. Man Cybern. Part. B (Cybern.)"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Chatterjee, A., and Yuen, P.W. (2019). Endmember learning with k-means through scd model in hyperspectral scene reconstructions. J. Imaging, 5.","DOI":"10.3390\/jimaging5110085"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"2388","DOI":"10.1109\/TNNLS.2020.3005234","article-title":"Deep Blind Hyperspectral Image Super-Resolution","volume":"32","author":"Zhang","year":"2020","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"3647","DOI":"10.1109\/TNNLS.2017.2736011","article-title":"Learning low-rank decomposition for pan-sharpening with spatial-spectral offsets","volume":"29","author":"Yang","year":"2017","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"5997","DOI":"10.1109\/TGRS.2017.2718728","article-title":"Hyperspectral and multispectral image fusion based on local low rank and coupled spectral unmixing","volume":"55","author":"Zhou","year":"2017","journal-title":"IEEE Trans. Geosci. Remote. Sens."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"808","DOI":"10.1109\/TGRS.2017.2755016","article-title":"A universal destriping framework combining 1-D and 2-D variational optimization methods","volume":"56","author":"Liu","year":"2017","journal-title":"IEEE Trans. Geosci. Remote. Sens."},{"key":"ref_27","unstructured":"Shen, H. (September, January 5). Integrated fusion method for multiple temporal-spatial-spectral images. Proceedings of the 22nd ISPRS Congress, Melbourne, Australia."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Han, X.-H., Zheng, Y., and Chen, Y.-W. (2019, January 27\u201328). Multi-level and multi-scale spatial and spectral fusion CNN for hyperspectral image super-resolution. Proceedings of the IEEE\/CVF International Conference on Computer Vision Workshops, Seoul, Korea.","DOI":"10.1109\/ICCVW.2019.00533"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"639","DOI":"10.1109\/LGRS.2017.2668299","article-title":"Multispectral and hyperspectral image fusion using a 3-D-convolutional neural network","volume":"14","author":"Palsson","year":"2017","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Mei, S., Yuan, X., Ji, J., Zhang, Y., Wan, S., and Du, Q. (2017). Hyperspectral image spatial super-resolution via 3D full convolutional neural network. Remote. Sens., 9.","DOI":"10.3390\/rs9111139"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Xie, Q., Zhou, M., Zhao, Q., Meng, D., Zuo, W., and Xu, Z. (2019, January 15\u201320). Multispectral and hyperspectral image fusion by MS\/HS fusion net. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00168"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"7373","DOI":"10.1109\/JSTARS.2021.3097178","article-title":"UMAG-Net: A New Unsupervised Multi-attention-guided Network for Hyperspectral and Multispectral Image Fusion","volume":"14","author":"Liu","year":"2021","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"6815","DOI":"10.1109\/TIP.2021.3098246","article-title":"Cascaded Convolutional Neural Network-Based Hyperspectral Image Resolution Enhancement via an Auxiliary Panchromatic Image","volume":"30","author":"Lu","year":"2021","journal-title":"IEEE Trans. Image Process."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Gatys, L.A., Ecker, A.S., and Bethge, M. (2016, January 27\u201330). Image style transfer using convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.265"},{"key":"ref_35","unstructured":"Ulyanov, D., Vedaldi, A., and Lempitsky, V. (2018, January 18\u201323). Deep image prior. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA."},{"key":"ref_36","unstructured":"Shaham, T.R., Dekel, T., and Michaeli, T. (November, January 27). Singan: Learning a generative model from a single natural image. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Seoul, Korea."},{"key":"ref_37","unstructured":"Shocher, A., Bagon, S., Isola, P., and Irani, M. (November, January 27). Ingan: Capturing and retargeting the \"DNA\" of a natural image. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Seoul, Korea."},{"key":"ref_38","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv Prepr."},{"key":"ref_39","first-page":"262","article-title":"Texture synthesis using convolutional neural networks","volume":"28","author":"Gatys","year":"2015","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"3008","DOI":"10.1109\/JSTARS.2015.2440092","article-title":"Hyper-sharpening: A first approach on SIM-GA data","volume":"8","author":"Selva","year":"2015","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"3373","DOI":"10.1109\/TGRS.2014.2375320","article-title":"A convex formulation for hyperspectral image superresolution via subspace-based regularization","volume":"53","author":"Simoes","year":"2014","journal-title":"IEEE Trans. Geosci. Remote. Sens."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/13\/16\/3226\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:45:54Z","timestamp":1760165154000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/13\/16\/3226"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,13]]},"references-count":41,"journal-issue":{"issue":"16","published-online":{"date-parts":[[2021,8]]}},"alternative-id":["rs13163226"],"URL":"https:\/\/doi.org\/10.3390\/rs13163226","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,8,13]]}}}