{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T02:52:59Z","timestamp":1760151179724,"version":"build-2065373602"},"reference-count":47,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2022,3,3]],"date-time":"2022-03-03T00:00:00Z","timestamp":1646265600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Hyperspectral images (HSIs) are data cubes containing rich spectral information, making them beneficial to many Earth observation missions. However, due to the limitations of the associated imaging systems and their sensors, such as the swath width and revisit period, hyperspectral imagery over a large coverage area cannot be acquired in a short amount of time. Spectral super-resolution (SSR) is a method that involves learning the relationship between a multispectral image (MSI) and an HSI, based on the overlap region, followed by reconstruction of the HSI by making full use of the large swath width of the MSI, thereby improving its coverage. Much research has been conducted recently to address this issue, but most existing methods mainly learn the prior spectral information from training data, lacking constraints on the resulting spectral fidelity. To address this problem, a novel learning spectral transformer network (LSTNet) is proposed in this paper, utilizing a reference-based learning strategy to transfer the spectral structure knowledge of a reference HSI to create a reasonable reconstruction spectrum. More specifically, a spectral transformer module (STM) and a spectral reconstruction module (SRM) are designed, in order to exploit the prior and reference spectral information. Experimental results demonstrate that the proposed method has the ability to produce high-fidelity reconstructed spectra.<\/jats:p>","DOI":"10.3390\/s22051978","type":"journal-article","created":{"date-parts":[[2022,3,3]],"date-time":"2022-03-03T20:36:30Z","timestamp":1646339790000},"page":"1978","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["LSTNet: A Reference-Based Learning Spectral Transformer Network for Spectral Super-Resolution"],"prefix":"10.3390","volume":"22","author":[{"given":"Debao","family":"Yuan","sequence":"first","affiliation":[{"name":"School of Geoscience and Surveying Engineering, China University of Mining and Technology (Beijing), Beijing 100083, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ling","family":"Wu","sequence":"additional","affiliation":[{"name":"School of Geoscience and Surveying Engineering, China University of Mining and Technology (Beijing), Beijing 100083, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Huinan","family":"Jiang","sequence":"additional","affiliation":[{"name":"School of Geoscience and Surveying Engineering, China University of Mining and Technology (Beijing), Beijing 100083, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bingrui","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Geoscience and Surveying Engineering, China University of Mining and Technology (Beijing), Beijing 100083, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jian","family":"Li","sequence":"additional","affiliation":[{"name":"School of Geoscience and Surveying Engineering, China University of Mining and Technology (Beijing), Beijing 100083, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,3,3]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1109\/79.974724","article-title":"Detection Algorithms for Hyperspectral Imaging Applications","volume":"19","author":"Manolakis","year":"2002","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"999","DOI":"10.1080\/10408398.2010.543495","article-title":"Principles and Applications of Hyperspectral Imaging in Quality Evaluation of Agro-Food Products: A Review","volume":"52","author":"Elmasry","year":"2012","journal-title":"Crit. Rev. Food Sci. Nutr."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s11947-016-1817-8","article-title":"Extraction of Spectral Information from Hyperspectral Data and Application of Hyperspectral Imaging for Food and Agricultural Products","volume":"10","author":"Ravikanth","year":"2017","journal-title":"Food Bioprocess Technol."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"10901","DOI":"10.1117\/1.JBO.19.1.010901","article-title":"Medical Hyperspectral Imaging: A Review","volume":"19","author":"Lu","year":"2014","journal-title":"J. Biomed. Opt."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"118","DOI":"10.1109\/MGRS.2019.2911100","article-title":"Hyperspectral Band Selection: A Review","volume":"7","author":"Sun","year":"2019","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1109\/MGRS.2021.3064051","article-title":"Interpretable Hyperspectral Artificial Intelligence: When Nonconvex Modeling Meets Hyperspectral Remote Sensing","volume":"9","author":"Hong","year":"2021","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"2487","DOI":"10.1109\/TGRS.2020.3006534","article-title":"Coupled Convolutional Neural Network with Adaptive Response Function Learning for Unsupervised Hyperspectral Super Resolution","volume":"59","author":"Zheng","year":"2021","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"2142","DOI":"10.1109\/LGRS.2017.2755061","article-title":"Multiscale Superpixel-Level Subspace-Based Support Vector Machines for Hyperspectral Image Classification","volume":"14","author":"Yu","year":"2017","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_9","first-page":"1","article-title":"Spectral-Spatial Feature Tokenization Transformer for Hyperspectral Image Classification","volume":"13","author":"Sun","year":"2022","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"4233","DOI":"10.1109\/TGRS.2020.3024852","article-title":"Target Detection Through Tree-Structured Encoding for Hyperspectral Images","volume":"59","author":"Sun","year":"2021","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1109\/MGRS.2016.2637824","article-title":"Hyperspectral and Multispectral Data Fusion: A Comparative Review of the Recent Literature","volume":"5","author":"Yokoya","year":"2017","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_12","first-page":"5520216","article-title":"NonRegSRNet: A Non-Rigid Registration Hyperspectral Super-Resolution Network","volume":"59","author":"Zheng","year":"2021","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"2269","DOI":"10.1109\/TGRS.2020.3000684","article-title":"Spectral Superresolution of Multispectral Imagery with Joint Sparse and Low-Rank Learning","volume":"59","author":"Gao","year":"2021","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Li, K., Dai, D., and Van Gool, L. (2022, January 3\u20138). Hyperspectral Image Super-Resolution with RGB Image Super-Resolution as an Auxiliary Task. Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.","DOI":"10.1109\/WACV51458.2022.00409"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1016\/j.neunet.2021.11.014","article-title":"Dilated Projection Correction Network Based on Autoencoder for Hyperspectral Image Super-Resolution","volume":"146","author":"Wang","year":"2022","journal-title":"Neural Netw."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"4650","DOI":"10.1109\/TGRS.2018.2832228","article-title":"A Deep Network Architecture for Super-Resolution-Aided Hyperspectral Image Classification with Classwise Loss","volume":"56","author":"Hao","year":"2018","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"2593","DOI":"10.1007\/s13042-018-00911-4","article-title":"Separable-Spectral Convolution and Inception Network for Hyperspectral Image Super-Resolution","volume":"10","author":"Zheng","year":"2019","journal-title":"Int. J. Mach. Learn. Cybern."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"140302","DOI":"10.1007\/s11432-019-2805-y","article-title":"A New Sensor Bias-Driven Spatio-Temporal Fusion Model Based on Convolutional Neural Networks","volume":"63","author":"Li","year":"2020","journal-title":"Sci. China Inf. Sci."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Arad, B., and Ben-Shahar, O. (2016, January 11\u201314). Sparse Recovery of Hyperspectral Signal from Natural RGB Images. Proceedings of the Computer Vision\u2014ECCV 2016, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46478-7_2"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Nguyen, R.M.H., Prasad, D.K., and Brown, M.S. (2014, January 6\u201312). Training-Based Spectral Reconstruction from a Single RGB Image. Proceedings of the Computer Vision\u2014ECCV 2014, Zurich, Switzerland.","DOI":"10.1007\/978-3-319-10584-0_13"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Jia, Y., Zheng, Y., Gu, L., Subpa-Asa, A., Lam, A., Sato, Y., and Sato, I. (2017, January 22\u201329). From RGB to Spectrum for Natural Scenes via Manifold-Based Mapping. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.504"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Xiong, Z., Shi, Z., Li, H., Wang, L., Liu, D., and Wu, F. (2017, January 22\u201329). Hscnn: Cnn-Based Hyperspectral Image Recovery from Spectrally Undersampled Projections. Proceedings of the IEEE International Conference on Computer Vision Workshops, Venice, Italy.","DOI":"10.1109\/ICCVW.2017.68"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Shi, Z., Chen, C., Xiong, Z., Liu, D., and Wu, F. (2018, January 18\u201322). Hscnn+: Advanced Cnn-Based Hyperspectral Recovery from Rgb Images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPRW.2018.00139"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Stiebel, T., Koppers, S., Seltsam, P., and Merhof, D. (2018, January 18\u201322). Reconstructing Spectral Images from Rgb-Images Using a Convolutional Neural Network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPRW.2018.00140"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Zhang, L., Lang, Z., Wang, P., Wei, W., Liao, S., Shao, L., and Zhang, Y. (2020, January 7\u201312). Pixel-Aware Deep Function-Mixture Network for Spectral Super-Resolution. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.","DOI":"10.1609\/aaai.v34i07.6978"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Li, J., Wu, C., Song, R., Li, Y., and Liu, F. (2020, January 14\u201319). Adaptive Weighted Attention Network with Camera Spectral Sensitivity Prior for Spectral Reconstruction from RGB Images. Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle, WA, USA.","DOI":"10.1109\/CVPRW50498.2020.00239"},{"key":"ref_27","unstructured":"Li, J., Wu, C., Song, R., Li, Y., Xie, W., He, L., and Gao, X. (2021). Deep Hybrid 2-D-3-D CNN Based on Dual Second-Order Attention with Camera Spectral Sensitivity Prior for Spectral Super-Resolution. IEEE Trans. Neural Netw. Learn. Syst., 1\u201312."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"7256","DOI":"10.1109\/TIP.2021.3104177","article-title":"Spectral Super-Resolution Network Guided by Intrinsic Properties of Hyperspectral Imagery","volume":"30","author":"Hang","year":"2021","journal-title":"IEEE Trans. Image Process."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1016\/j.inffus.2021.10.016","article-title":"PoNet: A Universal Physical Optimization-Based Spectral Super-Resolution Network for Arbitrary Multispectral Images","volume":"80","author":"He","year":"2022","journal-title":"Inf. Fusion"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"2011","DOI":"10.1109\/TPAMI.2019.2913372","article-title":"Squeeze-and-Excitation Networks","volume":"42","author":"Hu","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Zhang, H., Dana, K., Shi, J., Zhang, Z., Wang, X., Tyagi, A., and Agrawal, A. (2018, January 18\u201322). Context Encoding for Semantic Segmentation. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00747"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., and Fu, Y. (2018, January 8\u201314). Image Super-Resolution Using Very Deep Residual Channel Attention Networks. Proceedings of the European conference on computer vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01234-2_18"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Yang, Z., Zhu, L., Wu, Y., and Yang, Y. (2020, January 14\u201319). Gated Channel Transformation for Visual Recognition. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01181"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Dai, T., Cai, J., Zhang, Y., Xia, S.-T., and Zhang, L. (2019, January 15\u201320). Second-Order Attention Network for Single Image Super-Resolution. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.01132"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Qin, Z., Zhang, P., Wu, F., and Li, X. (2021, January 11\u201317). Fcanet: Frequency Channel Attention Networks. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Montreal, BC, Canada.","DOI":"10.1109\/ICCV48922.2021.00082"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Liu, M., Wang, Z., and Ji, S. (2021). Non-Local Graph Neural Networks. IEEE Trans. Pattern Anal. Mach. Intell., 1.","DOI":"10.1109\/TPAMI.2021.3081499"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Hu, H., Gu, J., Zhang, Z., Dai, J., and Wei, Y. (2018, January 18\u201322). Relation Networks for Object Detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00378"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Chen, Y., Rohrbach, M., Yan, Z., Shuicheng, Y., Feng, J., and Kalantidis, Y. (2019, January 15\u201320). Graph-Based Global Reasoning Networks. Proceedings of the 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00052"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020, January 23\u201328). End-to-End Object Detection with Transformers. Proceedings of the Computer Vision\u2014ECCV 2020, Glasgow, UK.","DOI":"10.1007\/978-3-030-58452-8_13"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Yang, F., Yang, H., Fu, J., Lu, H., and Guo, B. (2020, January 14\u201319). Learning Texture Transformer Network for Image Super-Resolution. Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00583"},{"key":"ref_41","first-page":"7354","article-title":"Self-Attention Generative Adversarial Networks","volume":"Volume 97","author":"Chaudhuri","year":"2019","journal-title":"Proceedings of the 36th International Conference on Machine Learning"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Woo, S., Park, J., Lee, J.-Y., and Kweon, I.S. (2018, January 8\u201314). Cbam: Convolutional Block Attention Module. Proceedings of the European conference on computer vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01234-2_1"},{"key":"ref_44","unstructured":"Wallach, H., Larochelle, H., Beygelzimer, A., d\u2019Alch\u00e9-Buc, F., Fox, E., and Garnett, R. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems 32, Curran Associates, Inc."},{"key":"ref_45","unstructured":"Kingma, D.P., and Ba, J.L. (2015, January 7\u20139). Adam: A method for Stochastic Optimization. Proceedings of the 3rd International Conference for Learning Representations, San Diego, CA, USA."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Yan, Y., Zhang, L., Li, J., Wei, W., and Zhang, Y. (2018, January 23\u201326). Accurate Spectral Super-Resolution from Single RGB Image Using Multi-Scale CNN. Proceedings of the Pattern Recognition and Computer Vision, Guangzhou, China.","DOI":"10.1007\/978-3-030-03335-4_18"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Zhao, Y., Po, L.-M., Yan, Q., Liu, W., and Lin, T. (2020, January 14\u201319). Hierarchical Regression Network for Spectral Reconstruction from RGB Images. Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle, WA, USA.","DOI":"10.1109\/CVPRW50498.2020.00219"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/5\/1978\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:31:24Z","timestamp":1760135484000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/5\/1978"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,3]]},"references-count":47,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2022,3]]}},"alternative-id":["s22051978"],"URL":"https:\/\/doi.org\/10.3390\/s22051978","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2022,3,3]]}}}