{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,5]],"date-time":"2026-02-05T13:46:33Z","timestamp":1770299193029,"version":"3.49.0"},"reference-count":42,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2024,3,19]],"date-time":"2024-03-19T00:00:00Z","timestamp":1710806400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"publisher","award":["3072022CF0801"],"award-info":[{"award-number":["3072022CF0801"]}],"id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"publisher","award":["2018YFE0206500"],"award-info":[{"award-number":["2018YFE0206500"]}],"id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"publisher","award":["614210202030217"],"award-info":[{"award-number":["614210202030217"]}],"id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100012166","name":"National Key R&amp;D Program of China","doi-asserted-by":"publisher","award":["3072022CF0801"],"award-info":[{"award-number":["3072022CF0801"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100012166","name":"National Key R&amp;D Program of China","doi-asserted-by":"publisher","award":["2018YFE0206500"],"award-info":[{"award-number":["2018YFE0206500"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100012166","name":"National Key R&amp;D Program of China","doi-asserted-by":"publisher","award":["614210202030217"],"award-info":[{"award-number":["614210202030217"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"publisher"}]},{"name":"National Key Laboratory of Communication Anti Jamming Technology","award":["3072022CF0801"],"award-info":[{"award-number":["3072022CF0801"]}]},{"name":"National Key Laboratory of Communication Anti Jamming Technology","award":["2018YFE0206500"],"award-info":[{"award-number":["2018YFE0206500"]}]},{"name":"National Key Laboratory of Communication Anti Jamming Technology","award":["614210202030217"],"award-info":[{"award-number":["614210202030217"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Utilizing multi-modal data, as opposed to only hyperspectral image (HSI), enhances target identification accuracy in remote sensing. Transformers are applied to multi-modal data classification for their long-range dependency but often overlook intrinsic image structure by directly flattening image blocks into vectors. Moreover, as the encoder deepens, unprofitable information negatively impacts classification performance. Therefore, this paper proposes a learnable transformer with an adaptive gating mechanism (AGMLT). Firstly, a spectral\u2013spatial adaptive gating mechanism (SSAGM) is designed to comprehensively extract the local information from images. It mainly contains point depthwise attention (PDWA) and asymmetric depthwise attention (ADWA). The former is for extracting spectral information of HSI, and the latter is for extracting spatial information of HSI and elevation information of LiDAR-derived rasterized digital surface models (LiDAR-DSM). By omitting linear layers, local continuity is maintained. Then, the layer Scale and learnable transition matrix are introduced to the original transformer encoder and self-attention to form the learnable transformer (L-Former). It improves data dynamics and prevents performance degradation as the encoder deepens. Subsequently, learnable cross-attention (LC-Attention) with the learnable transfer matrix is designed to augment the fusion of multi-modal data by enriching feature information. Finally, poly loss, known for its adaptability with multi-modal data, is employed in training the model. Experiments in the paper are conducted on four famous multi-modal datasets: Trento (TR), MUUFL (MU), Augsburg (AU), and Houston2013 (HU). The results show that AGMLT achieves optimal performance over some existing models.<\/jats:p>","DOI":"10.3390\/rs16061080","type":"journal-article","created":{"date-parts":[[2024,3,20]],"date-time":"2024-03-20T05:04:12Z","timestamp":1710911052000},"page":"1080","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Joint Classification of Hyperspectral and LiDAR Data Based on Adaptive Gating Mechanism and Learnable Transformer"],"prefix":"10.3390","volume":"16","author":[{"given":"Minhui","family":"Wang","sequence":"first","affiliation":[{"name":"Key Laboratory of Advanced Ship Communication and Information Technology, Harbin Engineering University, Harbin 150001, China"}]},{"given":"Yaxiu","family":"Sun","sequence":"additional","affiliation":[{"name":"Key Laboratory of Advanced Ship Communication and Information Technology, Harbin Engineering University, Harbin 150001, China"}]},{"given":"Jianhong","family":"Xiang","sequence":"additional","affiliation":[{"name":"Key Laboratory of Advanced Ship Communication and Information Technology, Harbin Engineering University, Harbin 150001, China"}]},{"given":"Rui","family":"Sun","sequence":"additional","affiliation":[{"name":"Key Laboratory of Advanced Ship Communication and Information Technology, Harbin Engineering University, Harbin 150001, China"}]},{"given":"Yu","family":"Zhong","sequence":"additional","affiliation":[{"name":"Agile and Intelligent Computing Key Laboratory, Chengdu 610000, China"}]}],"member":"1968","published-online":{"date-parts":[[2024,3,19]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Czaja, W., Kavalerov, I., and Li, W. (2021, January 24\u201326). Exploring the high dimensional geometry of HSI features. Proceedings of the 2021 11th Workshop on Hyperspectral Imaging and Signal Processing: Evolution in Remote Sensing (WHISPERS), Amsterdam, The Netherlands.","DOI":"10.1109\/WHISPERS52202.2021.9484048"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"641723","DOI":"10.3389\/frsen.2021.641723","article-title":"Challenges and opportunities in lidar remote sensing","volume":"2","author":"Wang","year":"2021","journal-title":"Front. Remote Sens."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"5516619","DOI":"10.1109\/TGRS.2021.3120198","article-title":"Revisiting deep hyperspectral feature extraction networks via gradient centralized convolution","volume":"60","author":"Roy","year":"2022","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1016\/j.rse.2015.05.023","article-title":"Measuring freshwater aquatic ecosystems: The need for a hyperspectral global mapping satellite mission","volume":"167","author":"Hestir","year":"2015","journal-title":"Remote Sens. Environ."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1109\/MGRS.2019.2902525","article-title":"Hyperspectral imaging for military and security applications: Combining myriad processing and sensing techniques","volume":"7","author":"Shimoni","year":"2019","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"364","DOI":"10.1109\/TIP.2022.3228497","article-title":"UIU-Net: U-Net in U-Net for infrared small object detection","volume":"32","author":"Wu","year":"2023","journal-title":"IEEE Trans. Image Process."},{"key":"ref_7","first-page":"287","article-title":"Hyper-spectral remote sensing applied to mineral exploration in southern peru:A multiple data integration approach in the chapi chiara gold prospect","volume":"64","author":"Carrino","year":"2018","journal-title":"Int. J. Appl. Earth Obs. Geoinf."},{"key":"ref_8","first-page":"2098759","article-title":"Review of Near Infrared Hyperspectral Imaging Applications Related to Wood and Wood Products","volume":"57","author":"Schimleck","year":"2022","journal-title":"Appl. Spectrosc. Rev."},{"key":"ref_9","first-page":"JTE20220073","article-title":"Rapeseed Storage Quality Detection Using Hyperspectral Image Technology\u2013An Application for Future Smart Cities","volume":"51","author":"Liao","year":"2022","journal-title":"J. Test. Eval."},{"key":"ref_10","first-page":"236","article-title":"Review of hyperspectral remote sensing image classification","volume":"20","author":"Du","year":"2016","journal-title":"J. Remote Sens."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"277","DOI":"10.1109\/LGRS.2019.2918719","article-title":"HybridSN: Exploring 3-D\u20132-D CNN feature hierarchy for hyperspectral image classification","volume":"17","author":"Roy","year":"2020","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"8702","DOI":"10.1109\/JSTARS.2023.3271901","article-title":"Heterogeneous spectral-spatial network with 3D attention and MLP for hyperspectral image classification using limited training samples","volume":"16","author":"Sun","year":"2023","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"5518615","DOI":"10.1109\/TGRS.2021.3130716","article-title":"SpectralFormer: Rethinking hyperspectral image classification with transformers","volume":"60","author":"Hong","year":"2022","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Sang, M., Zhao, Y., and Liu, G. (2023, January 4\u201310). Improving Transformer-Based Networks with Locality for Automatic Speaker Verification. Proceedings of the 2023 48th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.","DOI":"10.1109\/ICASSP49357.2023.10096333"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TGRS.2022.3231215","article-title":"Spectral\u2013spatial feature tokenization transformer for hyperspectral image classification","volume":"60","author":"Sun","year":"2022","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Wang, A., Xing, S., Zhao, Y., Wu, H., and Iwahori, Y. (2022). A hyperspectral image classification method based on adaptive spectral spatial kernel combined with improved vision transformer. Remote Sens., 14.","DOI":"10.3390\/rs14153705"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"809","DOI":"10.1109\/TGRS.2011.2162649","article-title":"Spectral\u2013spatial hyperspectral image segmentation using subspace multinomial logistic regression and Markov random fields","volume":"50","author":"Li","year":"2011","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_18","unstructured":"Qi, C.R., Yi, L., Su, H., and Guibas, L.J. (2017). PointNet++: Deep hierarchical feature learning on points a metric space. arXiv."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"856","DOI":"10.1109\/JSTSP.2012.2208177","article-title":"Classification of remote sensing optical and LiDAR data using extended attribute profiles","volume":"6","author":"Pedergnana","year":"2012","journal-title":"IEEE J. Sel. Top. Signal Process."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"3997","DOI":"10.1109\/TGRS.2017.2686450","article-title":"Hyperspectral and LiDAR fusion using extinction profiles and total variation component analysis","volume":"55","author":"Rasti","year":"2017","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1109\/MGRS.2017.2762307","article-title":"Deep learning in remote sensing: A comprehensive review and list of resources","volume":"5","author":"Zhu","year":"2017","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"5530416","DOI":"10.1109\/TGRS.2022.3177633","article-title":"Hyperspectral and LiDAR data classification using joint CNNs and morphological feature learning","volume":"60","author":"Roy","year":"2022","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"5704513","DOI":"10.1109\/TGRS.2023.3321057","article-title":"Hashing-based deep metric learning for the classification of hyperspectral and LiDAR data","volume":"61","author":"Song","year":"2023","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"937","DOI":"10.1109\/TGRS.2017.2756851","article-title":"Multisource remote sensing data classification based on convolutional neural network","volume":"56","author":"Xu","year":"2017","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"5541213","DOI":"10.1109\/TGRS.2022.3216319","article-title":"Global\u2013local transformer network for HSI and LiDAR data joint classification","volume":"60","author":"Ding","year":"2022","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1130","DOI":"10.1109\/JSTARS.2022.3232995","article-title":"Local Information interaction transformer for hyperspectral and LiDAR data classification","volume":"16","author":"Zhang","year":"2022","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Xu, H., Zheng, T., Liu, Y., Zhang, Z., Xue, C., and Li, J. (2024). A joint convolutional cross ViT network for hyperspectral and light detection and ranging fusion classification. Remote Sens., 16.","DOI":"10.3390\/rs16030489"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"5515620","DOI":"10.1109\/TGRS.2023.3286826","article-title":"Multimodal fusion transformer for remote sensing image classification","volume":"61","author":"Roy","year":"2023","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_29","first-page":"5500716","article-title":"Joint classification of hyperspectral and LiDAR data using a hierarchical CNN and transformer","volume":"61","author":"Zhao","year":"2023","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Wang, Y., Li, Y., Wang, G., and Liu, X. (2022). Multi-scale attention network for single image super-resolution. arXiv.","DOI":"10.1109\/ICPR56361.2022.9956541"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Gulati, A., Qin, J., and Chiu, C.C. (2020). Conformer: Convolution-augmented transformer for speech recognition. arXiv.","DOI":"10.21437\/Interspeech.2020-3015"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"4939","DOI":"10.1109\/TGRS.2020.2969024","article-title":"Classification of hyperspectral and LiDAR data using coupled CNNs","volume":"58","author":"Hang","year":"2020","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_33","unstructured":"Hendrycks, D., and Gimpel, K. (2016). Gaussian Error Linear Units (gelus). arXiv."},{"key":"ref_34","unstructured":"Zhou, D., Kang, B., Jin, X., and Yang, L. (2021). DeepViT: Towards deeper vision transformer. arXiv."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Touvron, H., Cord, M., and Sablayrolles, A. (2021). Going deeper with image transformers. arXiv.","DOI":"10.1109\/ICCV48922.2021.00010"},{"key":"ref_36","unstructured":"Leng, Z.Q., Tan, M.X., and Liu, C.X. (2022, January 25\u201329). PolyLoss: A polynomial expansion perspective of classification loss functions. Proceedings of the 2022 10th IEEE Conference on International Conference on Learning Representations (ICLR), Virtual."},{"key":"ref_37","unstructured":"Gader, P., Zare, A., Close, R., Aitken, J., and Tuell, G. (2013). Muufl Gulfport Hyperspectral and LiDAR Airborne Data Set, University of Florida. Technical Report REP-2013\u2013570."},{"key":"ref_38","unstructured":"Du, X., and Zare, A. (2017). Scene Label Ground Truth Map for Muufl Gulfport Data Set, University of Florida. Technical Report 20170417."},{"key":"ref_39","first-page":"371","article-title":"Characterisation methods for the hyperspectral sensor HySpex at DLR\u2019s calibration home base","volume":"8533","author":"Baumgartner","year":"2012","journal-title":"Proc. SPIE"},{"key":"ref_40","unstructured":"Kurz, F., Rosenbaum, D., Leitloff, J., Meynberg, O., and Reinartz, P. (2011, January 18\u201319). Real time camera system for disaster and traffic monitoring. Proceedings of International Conference on SMPR, Tehran, Iran."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"5511305","DOI":"10.1109\/LGRS.2021.3126125","article-title":"End-to-End Multilevel Hybrid Attention Framework for Hyperspectral Image Classification","volume":"19","author":"Xiang","year":"2022","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_42","first-page":"5503615","article-title":"Spectral\u2013spatial morphological attention transformer for hyperspectral image classification","volume":"61","author":"Swalpa","year":"2022","journal-title":"IEEE Trans. Geosci. Remote Sens."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/16\/6\/1080\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T14:16:13Z","timestamp":1760105773000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/16\/6\/1080"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,19]]},"references-count":42,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2024,3]]}},"alternative-id":["rs16061080"],"URL":"https:\/\/doi.org\/10.3390\/rs16061080","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,19]]}}}