{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,20]],"date-time":"2026-02-20T07:23:06Z","timestamp":1771572186579,"version":"3.50.1"},"reference-count":58,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2022,1,27]],"date-time":"2022-01-27T00:00:00Z","timestamp":1643241600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Grant-in Aid for Scientific Research from the Japanese Ministry for Education, Science, Culture and Sports","award":["20K11867"],"award-info":[{"award-number":["20K11867"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Multimedia Comput. Commun. Appl."],"published-print":{"date-parts":[[2022,1,31]]},"abstract":"<jats:p>Hyperspectral imaging is a promising imaging modality that simultaneously captures several images for the same scene on narrow spectral bands, and it has made considerable progress in different fields, such as agriculture, astronomy, and surveillance. However, the existing hyperspectral (HS) cameras sacrifice the spatial resolution for providing the detail spectral distribution of the imaged scene, which leads to low-resolution (LR) HS images compared with the common red-green-blue (RGB) images. Generating a high-resolution HS (HR-HS) image via fusing an observed LR-HS image with the corresponding HR-RGB image has been actively studied. Existing methods for this fusing task generally investigate hand-crafted priors to model the inherent structure of the latent HR-HS image, and they employ optimization approaches for solving it. However, proper priors for different scenes can possibly be diverse, and to figure it out for a specific scene is difficult. This study investigates a deep convolutional neural network (DCNN)-based method for automatic prior learning, and it proposes a novel fusion DCNN model with multi-scale spatial and spectral learning for effectively merging an HR-RGB and LR-HS images. Specifically, we construct an U-shape network architecture for gradually reducing the feature sizes of the HR-RGB image (Encoder-side) and increasing the feature sizes of the LR-HS image (Decoder-side), and we fuse the HR spatial structure and the detail spectral attribute in multiple scales for tackling the large resolution difference in spatial domain of the observed HR-RGB and LR-HS images. Then, we employ multi-level cost functions for the proposed multi-scale learning network to alleviate the gradient vanish problem in long-propagation procedure. In addition, for further improving the reconstruction performance of the HR-HS image, we refine the predicted HR-HS image using an alternating back-projection method for minimizing the reconstruction errors of the observed LR-HS and HR-RGB images. Experiments on three benchmark HS image datasets demonstrate the superiority of the proposed method in both quantitative values and visual qualities.<\/jats:p>","DOI":"10.1145\/3477396","type":"journal-article","created":{"date-parts":[[2022,1,27]],"date-time":"2022-01-27T19:44:21Z","timestamp":1643312661000},"page":"1-21","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Hyperspectral Image Reconstruction Using Multi-scale Fusion Learning"],"prefix":"10.1145","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5003-3180","authenticated-orcid":false,"given":"Xian-Hua","family":"Han","sequence":"first","affiliation":[{"name":"Graduate School of Science and Technology for Innovation, Yamaguchi University, Yoshida, Yamaguchi, Japan"}]},{"given":"Yinqiang","family":"Zheng","sequence":"additional","affiliation":[{"name":"National Institute of Informatics, Bunkyo-ku, Tokyo, Japan"}]},{"given":"Yen-Wei","family":"Chen","sequence":"additional","affiliation":[{"name":"College of Information Science and Engineering, Ritsumeikan University, Kusatsu, Shiga, Japan"}]}],"member":"320","published-online":{"date-parts":[[2022,1,27]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/LGRS.2008.2012003"},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICPR.2014.640"},{"key":"e_1_3_1_4_2","first-page":"63","article-title":"Sparse spatio-spectral representation for hyperspectral image super-resolution","author":"Akhtar N.","year":"2014","unstructured":"N. Akhtar, F. Shafait, and A. Mian. 2014. Sparse spatio-spectral representation for hyperspectral image super-resolution. In Proceedings of the ECCV. 63\u201378.","journal-title":"Proceedings of the ECCV"},{"key":"e_1_3_1_5_2","first-page":"3631","article-title":"Bayesian sparse representation for hyperspectral image super resolution","author":"Akhtar N.","year":"2015","unstructured":"N. Akhtar, F. Shafait, and A. Mian. 2015. Bayesian sparse representation for hyperspectral image super resolution. In Proceedings of the CVPR. 3631\u20133640.","journal-title":"Proceedings of the CVPR"},{"key":"e_1_3_1_6_2","article-title":"Adversarial networks for spatial context-aware spectral image reconstruction from RGB","author":"Alvarez-Gila Aitor","year":"2017","unstructured":"Aitor Alvarez-Gila, Joost van de Weijer, and Estibaliz Garrote. 2017. Adversarial networks for spatial context-aware spectral image reconstruction from RGB. In Proceedings of the ICCVW.","journal-title":"Proceedings of the ICCVW"},{"key":"e_1_3_1_7_2","first-page":"19","article-title":"Sparse recovery of hyperspectral signal from natural rgb images","author":"Arad B.","year":"2016","unstructured":"B. Arad and O. Ben-Shahar. 2016. Sparse recovery of hyperspectral signal from natural rgb images. In Proceedings of the ECCV. 19\u201334.","journal-title":"Proceedings of the ECCV"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/MGRS.2013.2244672"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1080\/01431160802639525"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2011.5995660"},{"issue":"7","key":"e_1_3_1_11_2","first-page":"1779","article-title":"Comparison of three different methods to merge multiresolution and multispectral data: Landsat TM and SPOT panchromatic","volume":"30","author":"Chavez P. S.","year":"1991","unstructured":"P. S. Chavez, S. C. Sides, and J. A. Anderson. 1991. Comparison of three different methods to merge multiresolution and multispectral data: Landsat TM and SPOT panchromatic. Photogramm. Eng. Rem. S. 30, 7 (1991), 1779\u20131804.","journal-title":"Photogramm. Eng. Rem. S."},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2019.2916734"},{"key":"e_1_3_1_13_2","first-page":"1","article-title":"Deep hyperspectral image sharpening","author":"Dian R. W.","year":"2018","unstructured":"R. W. Dian, S. T. Li, A. J. Guo, and L. Y. Fang. 2018. Deep hyperspectral image sharpening. IEEE Trans. Neural Netw. Learn. Syst. (2018), 1\u201311.","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2015.2439281"},{"key":"e_1_3_1_15_2","article-title":"Accelerating the super-resolution convolutional neural network","author":"Dong C.","year":"2016","unstructured":"C. Dong, C. C. Loy, and X. O. Tang. 2016. Accelerating the super-resolution convolutional neural network. In Proceedings of the ECCV.","journal-title":"Proceedings of the ECCV"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2018.2855412"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2012.2197589"},{"key":"e_1_3_1_18_2","unstructured":"S. Galliani C. Lanaras D. Marmanis E. Baltsavias and K. Schindler. 2017. Learned spectral super-resolution. Retrieved from https:\/\/arXiv:1703.09470."},{"key":"e_1_3_1_19_2","article-title":"Jointly sparse fusion of hyperspectral and multispectral imagery","author":"Grohnfeldt C.","year":"2013","unstructured":"C. Grohnfeldt, X. X. Zhu, and R. Bamler. 2013. Jointly sparse fusion of hyperspectral and multispectral imagery. In Proceedings of the IGARSS.","journal-title":"Proceedings of the IGARSS"},{"key":"e_1_3_1_20_2","first-page":"266","article-title":"Deep residual network of spectral and spatial fusion for hyperspectral image super-resolutionn","author":"Han X.-H.","year":"2019","unstructured":"X.-H. Han and Y.-W. Chen. 2019. Deep residual network of spectral and spatial fusion for hyperspectral image super-resolutionn. In Proceedings of the BigMM. 266\u2013270.","journal-title":"Proceedings of the BigMM"},{"key":"e_1_3_1_21_2","first-page":"2664","article-title":"Residual HSRCNN: Residual hyper-spectral reconstruction CNN from an RGB image","author":"Han X.-H.","year":"2018","unstructured":"X.-H. Han, B. X. Shi, and Y. Q. Zheng. 2018. Residual HSRCNN: Residual hyper-spectral reconstruction CNN from an RGB image. In Proceedings of the ICPR. 2664\u20132669.","journal-title":"Proceedings of the ICPR"},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2018.2855418"},{"key":"e_1_3_1_23_2","first-page":"2506","article-title":"SSF-CNN: Spatial and spectral fusion with CNN for hyperspectral image super-resolution","author":"Han X.-H.","year":"2018","unstructured":"X.-H. Han, B. X. Shi, and Y. Q. Zheng. 2018. SSF-CNN: Spatial and spectral fusion with CNN for hyperspectral image super-resolution. In Proceedings of the ICIP. 2506\u20132510.","journal-title":"Proceedings of the ICIP"},{"key":"e_1_3_1_24_2","first-page":"4330","article-title":"Multi-level and multi-scale spatial and spectral fusion CNN for hyperspectral image super-resolution","author":"Han X.-H.","year":"2019","unstructured":"X.-H. Han, Y. Q. Zheng, and Y.-W. Chen. 2019. Multi-level and multi-scale spatial and spectral fusion CNN for hyperspectral image super-resolution. In Proceedings of the ICCV Workshop. 4330\u20134339.","journal-title":"Proceedings of the ICCV Workshop"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11760-016-0880-y"},{"key":"e_1_3_1_26_2","article-title":"Application of the IHS color transform to the processing of multisensor data and image enhancement","author":"Haydn R.","year":"1982","unstructured":"R. Haydn, G. W. Dalke, J. Henkel, and J. E. Bare. 1982. Application of the IHS color transform to the processing of multisensor data and image enhancement. In Proceedings of the ISRSE.","journal-title":"Proceedings of the ISRSE"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/TGRS.2013.2253612"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/2647868.2654889"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2011.5995457"},{"key":"e_1_3_1_30_2","article-title":"Accurate image super-resolution using very deep convolutional networks","author":"Kim J.","year":"2016","unstructured":"J. Kim, J. K. Lee, and K. M. Lee. 2016. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the CVPR.","journal-title":"Proceedings of the CVPR"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1016\/0034-4257(93)90013-N"},{"key":"e_1_3_1_32_2","first-page":"6180","article-title":"Resolution enhancement for hyperspectral images: A super-resolution and fusion approach","author":"Kwan C.","year":"2017","unstructured":"C. Kwan, J. H. Cjoi, S. H. Chan, J. Zhou, and B. Budavari. 2017. Resolution enhancement for hyperspectral images: A super-resolution and fusion approach. In Proceedings of the ICASSP. 6180\u20136184.","journal-title":"Proceedings of the ICASSP"},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.3390\/rs10091416"},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.409"},{"key":"e_1_3_1_35_2","article-title":"Photo-realistic single image super-resolution using a generative adversarial network","author":"Ledig C.","year":"2017","unstructured":"C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Z. Shi. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the CVPR.","journal-title":"Proceedings of the CVPR"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.5555\/3008751.3008829"},{"issue":"3","key":"e_1_3_1_37_2","first-page":"171","article-title":"Deep learning for image denoising","volume":"7","author":"Li H. M.","year":"2014","unstructured":"H. M. Li. 2014. Deep learning for image denoising. Int. J. Signal Process. Image Process. Pattern Recogn. 7, 3 (2014), 171\u2013180.","journal-title":"Int. J. Signal Process. Image Process. Pattern Recogn."},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2017.05.024"},{"issue":"11","key":"e_1_3_1_39_2","article-title":"Hyperspectral image spatial super-resolution via 3D full convolutional neural network","volume":"9","author":"Mei S. H.","year":"2017","unstructured":"S. H. Mei, X. Yuan, J. Y. Ji, and Y. F. Zhang. 2017. Hyperspectral image spatial super-resolution via 3D full convolutional neural network. Neurocomputing 9, 11 (2017).","journal-title":"Neurocomputing"},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/LGRS.2005.861699"},{"key":"e_1_3_1_41_2","first-page":"44","article-title":"Tracking via object reflectance using a hyperspectral video camera","author":"Nguyen H. V.","year":"2010","unstructured":"H. V. Nguyen, A. Benerjee, and R. Chellappa. 2010. Tracking via object reflectance using a hyperspectral video camera. In Proceedings of the CVPRW. 44\u201351.","journal-title":"Proceedings of the CVPRW"},{"key":"e_1_3_1_42_2","first-page":"186","article-title":"Training-based spectral reconstruction from a single RGB image","author":"Nguyen R. M.","year":"2014","unstructured":"R. M. Nguyen, D. K. Prasad, and M. S. Brown. 2014. Training-based spectral reconstruction from a single RGB image. In Proceedings of the ECCV. 186\u2013201.","journal-title":"Proceedings of the ECCV"},{"key":"e_1_3_1_43_2","first-page":"2403","article-title":"DeepID-Net: Deformable deep convolu- tional neural networks for object detection","author":"Ouyang W.","year":"2015","unstructured":"W. Ouyang, X. Wang, X. Zeng, and S. Qiu. 2015. DeepID-Net: Deformable deep convolu- tional neural networks for object detection. In Proceedings of the CVPR. 2403\u20132412.","journal-title":"Proceedings of the CVPR"},{"key":"e_1_3_1_44_2","first-page":"2862","article-title":"Unsupervised sparse dirichlet-net for hyperspectral image super-resolution","author":"Qu Y.","year":"2018","unstructured":"Y. Qu, H. Qi, and C. Kwan. 2018. Unsupervised sparse dirichlet-net for hyperspectral image super-resolution. In Proceedings of the CVPR. 2862\u20132869.","journal-title":"Proceedings of the CVPR"},{"key":"e_1_3_1_45_2","unstructured":"K. Simonyan and A. Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. Retrieved from https:\/\/arxiv.org\/abs\/1409.1556."},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.5555\/2969033.2969049"},{"key":"e_1_3_1_47_2","unstructured":"C. Szegedy S. Reed D. Erhan D. Anguelov and S. Ioffe. 2015. Scalable high-quality object detection. Retrieved from https:\/\/arXiv:1412.1441v3. 1\u201310."},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCB.2009.2037132"},{"key":"e_1_3_1_49_2","first-page":"57.1\u201357.10","article-title":"Hyperspectral face recognition using 3D-DCT and partial least squares","author":"Uzair M.","year":"2013","unstructured":"M. Uzair, A. Mahmood, and A. Mian. 2013. Hyperspectral face recognition using 3D-DCT and partial least squares. In Proceedings of the BMVC. 57.1\u201357.10.","journal-title":"Proceedings of the BMVC"},{"key":"e_1_3_1_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2003.819861"},{"key":"e_1_3_1_51_2","first-page":"1409","article-title":"A non-negative sparse promoting algorithm for high-resolution hyperspectral imaging","author":"Wcoff E.","year":"2013","unstructured":"E. Wcoff, T. H. Chan, K. Jia, W. K. Ma, and Y. Ma. 2013. A non-negative sparse promoting algorithm for high-resolution hyperspectral imaging. In Proceedings of the ICASSP. 1409\u20131413.","journal-title":"Proceedings of the ICASSP"},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/TGRS.2014.2381272"},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2019.2893530"},{"key":"e_1_3_1_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/TGRS.2011.2161320"},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/2071389.2071391"},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2018.2862629"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1117\/1.JRS.10.035024"},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.394"},{"key":"e_1_3_1_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/LGRS.2008.919685"}],"container-title":["ACM Transactions on Multimedia Computing, Communications, and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3477396","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3477396","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:30:32Z","timestamp":1750188632000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3477396"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,27]]},"references-count":58,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,1,31]]}},"alternative-id":["10.1145\/3477396"],"URL":"https:\/\/doi.org\/10.1145\/3477396","relation":{},"ISSN":["1551-6857","1551-6865"],"issn-type":[{"value":"1551-6857","type":"print"},{"value":"1551-6865","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,27]]},"assertion":[{"value":"2020-09-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-07-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-01-27","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}