{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,7]],"date-time":"2026-05-07T16:30:29Z","timestamp":1778171429691,"version":"3.51.4"},"reference-count":45,"publisher":"MDPI AG","issue":"16","license":[{"start":{"date-parts":[[2022,8,19]],"date-time":"2022-08-19T00:00:00Z","timestamp":1660867200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62071466"],"award-info":[{"award-number":["62071466"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["6142A010402"],"award-info":[{"award-number":["6142A010402"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["2018GXNSFBA281086"],"award-info":[{"award-number":["2018GXNSFBA281086"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Fund of National Key Laboratory of Science and Technology on Remote Sensing Information and Imagery Analysis, Beijing Research Institute of Uranium Geology","award":["62071466"],"award-info":[{"award-number":["62071466"]}]},{"name":"Fund of National Key Laboratory of Science and Technology on Remote Sensing Information and Imagery Analysis, Beijing Research Institute of Uranium Geology","award":["6142A010402"],"award-info":[{"award-number":["6142A010402"]}]},{"name":"Fund of National Key Laboratory of Science and Technology on Remote Sensing Information and Imagery Analysis, Beijing Research Institute of Uranium Geology","award":["2018GXNSFBA281086"],"award-info":[{"award-number":["2018GXNSFBA281086"]}]},{"name":"Guangxi Natural Science Foundation","award":["62071466"],"award-info":[{"award-number":["62071466"]}]},{"name":"Guangxi Natural Science Foundation","award":["6142A010402"],"award-info":[{"award-number":["6142A010402"]}]},{"name":"Guangxi Natural Science Foundation","award":["2018GXNSFBA281086"],"award-info":[{"award-number":["2018GXNSFBA281086"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Multi-source image fusion is very important for improving image representation ability since its essence relies on the complementarity between multi-source information. However, feature-level image fusion methods based on the convolution neural network are impacted by the spatial misalignment between image pairs, which leads to the semantic bias in merging features and destroys the representation ability of the region-of-interests. In this paper, a novel multi-source image fusion transformer (MsIFT) is proposed. Due to the inherent global attention mechanism of the transformer, the MsIFT has non-local fusion receptive fields, and it is more robust to spatial misalignment. Furthermore, multiple classification-based downstream tasks (e.g., pixel-wise classification, image-wise classification and semantic segmentation) are unified in the proposed MsIFT framework, and the fusion module architecture is shared by different tasks. The MsIFT achieved state-of-the-art performances on the image-wise classification dataset VAIS, semantic segmentation dataset SpaceNet 6 and pixel-wise classification dataset GRSS-DFC-2013. The code and trained model are being released upon the publication of the work.<\/jats:p>","DOI":"10.3390\/rs14164062","type":"journal-article","created":{"date-parts":[[2022,8,22]],"date-time":"2022-08-22T01:56:40Z","timestamp":1661133400000},"page":"4062","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["MsIFT: Multi-Source Image Fusion Transformer"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2901-2593","authenticated-orcid":false,"given":"Xin","family":"Zhang","sequence":"first","affiliation":[{"name":"National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China"},{"name":"School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 101408, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hangzhi","family":"Jiang","sequence":"additional","affiliation":[{"name":"National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China"},{"name":"School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 101408, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nuo","family":"Xu","sequence":"additional","affiliation":[{"name":"National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China"},{"name":"School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 101408, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lei","family":"Ni","sequence":"additional","affiliation":[{"name":"Beijing Institute of Remote Sensing, Beijing 100085, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chunlei","family":"Huo","sequence":"additional","affiliation":[{"name":"National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China"},{"name":"School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 101408, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chunhong","family":"Pan","sequence":"additional","affiliation":[{"name":"National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China"},{"name":"School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 101408, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,8,19]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"937","DOI":"10.1109\/TGRS.2017.2756851","article-title":"Multisource remote sensing data classification based on convolutional neural network","volume":"56","author":"Xu","year":"2017","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"8891018","DOI":"10.1155\/2020\/8891018","article-title":"Dual-band maritime imagery ship classification based on multilayer convolutional feature fusion","volume":"2020","author":"Qiu","year":"2020","journal-title":"J. Sens."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"4237","DOI":"10.1109\/TGRS.2019.2961947","article-title":"Spatial\u2013spectral feature extraction via deep ConvLSTM neural networks for hyperspectral image classification","volume":"58","author":"Hu","year":"2020","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"738","DOI":"10.1109\/JSTARS.2020.2968930","article-title":"Robust capsule network based on maximum correntropy criterion for hyperspectral image classification","volume":"13","author":"Li","year":"2020","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_5","unstructured":"Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020, January 26\u201330). An Image is Worth 16 \u00d7 16 Words: Transformers for Image Recognition at Scale. Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020, January 23\u201328). End-to-end object detection with transformers. Proceedings of the European Conference on Computer Vision, Glasgow, UK.","DOI":"10.1007\/978-3-030-58452-8_13"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., and Torr, P.H. (2021, January 20\u201325). Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00681"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Chen, X., Yan, B., Zhu, J., Wang, D., Yang, X., and Lu, H. (2021, January 20\u201325). Transformer tracking. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00803"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Zhang, M.M., Choi, J., Daniilidis, K., Wolf, M.T., and Kanan, C. (2015, January 7\u201312). VAIS: A dataset for recognizing maritime imagery in the visible and infrared spectrums. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Boston, MA, USA.","DOI":"10.1109\/CVPRW.2015.7301291"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Shermeyer, J., Hogan, D., Brown, J., Van Etten, A., Weir, N., Pacifici, F., Hansch, R., Bastidas, A., Soenen, S., and Bacastow, T. (2020, January 14\u201319). SpaceNet 6: Multi-sensor all weather mapping dataset. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, WA, USA.","DOI":"10.1109\/CVPRW50498.2020.00106"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1109\/MGRS.2013.2277532","article-title":"Report on the 2013 IEEE GRSS data fusion contest: Fusion of hyperspectral and LiDAR data [technical committees]","volume":"1","author":"Pacifici","year":"2013","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Aziz, K., and Bouchara, F. (2018, January 27\u201329). Multimodal deep learning for robust recognizing maritime imagery in the visible and infrared spectrums. Proceedings of the International Conference Image Analysis and Recognition, P\u00f3voa de Varzim, Portugal.","DOI":"10.1007\/978-3-319-93000-8_27"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Santos, C.E., and Bhanu, B. (2018, January 7\u201310). Dyfusion: Dynamic IR\/RGB fusion for maritime vessel recognition. Proceedings of the 2018 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece.","DOI":"10.1109\/ICIP.2018.8451745"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"4340","DOI":"10.1109\/TGRS.2020.3016820","article-title":"More diverse means better: Multimodal deep learning meets remote-sensing imagery classification","volume":"59","author":"Hong","year":"2020","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1016\/j.inffus.2020.12.008","article-title":"A spatial-channel progressive fusion ResNet for remote sensing classification","volume":"70","author":"Zhu","year":"2021","journal-title":"Inf. Fusion"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"2971","DOI":"10.1109\/JSTARS.2015.2432037","article-title":"Fusion of hyperspectral and LiDAR remote sensing data using multiple feature learning","volume":"8","author":"Khodadadzadeh","year":"2015","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_17","unstructured":"Li, W., Gao, Y., Zhang, M., Tao, R., and Du, Q. (2022). Asymmetric Feature Fusion Network for Hyperspectral and SAR Image Classification. IEEE Trans. Neural Netw. Learn. Syst., 1\u201314."},{"key":"ref_18","first-page":"1","article-title":"Information Fusion for Classification of Hyperspectral and LiDAR Data Using IP-CNN","volume":"60","author":"Zhang","year":"2022","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Mohlax, S., Pande, S., Banerjee, B., and Chaudhuri, S. (2020, January 14\u201319). Fusatnet: Dual attention based spectrospatial multimodal fusion network for hyperspectral and lidar classification. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, WA, USA.","DOI":"10.21203\/rs.3.rs-32802\/v1"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1016\/j.inffus.2020.08.013","article-title":"Integrated fusion framework based on semicoupled sparse tensor factorization for spatio-temporal\u2013spectral fusion of remote sensing images","volume":"65","author":"Peng","year":"2021","journal-title":"Inf. Fusion"},{"key":"ref_21","first-page":"1","article-title":"ACLNN: Spatial, Spectral and Multiscale Attention ConvLSTM Neural Network for Multisource Remote Sensing Data Classification","volume":"33","author":"Li","year":"2020","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1016\/j.isprsjprs.2020.06.014","article-title":"X-ModalNet: A semi-supervised deep cross-modal network for classification of remote sensing data","volume":"167","author":"Hong","year":"2020","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1109\/TCYB.2018.2864670","article-title":"Feature Extraction for Classification of Hyperspectral and LiDAR Data Using Patch-to-Patch CNN","volume":"50","author":"Zhang","year":"2020","journal-title":"IEEE Trans. Cybern."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Huang, Z., Cheng, G., Wang, H., Li, H., Shi, L., and Pan, C. (2016, January 10\u201315). Building extraction from multi-source remote sensing images via deep deconvolution neural networks. Proceedings of the 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), Beijing, China.","DOI":"10.1109\/IGARSS.2016.7729471"},{"key":"ref_25","unstructured":"Liao, W., Bellens, R., Pi\u017eurica, A., Gautama, S., and Philips, W. (2014, January 13\u201318). Combining feature fusion and decision fusion for classification of hyperspectral and LiDAR data. Proceedings of the 2014 IEEE Geoscience and Remote Sensing Symposium, Quebec City, QC, Canada."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"4939","DOI":"10.1109\/TGRS.2020.2969024","article-title":"Classification of hyperspectral and LiDAR data using coupled CNNs","volume":"58","author":"Hang","year":"2020","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_27","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, \u0141., and Polosukhin, I. (2017, January 4\u20137). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., Ma, S., Xu, C., Xu, C., and Gao, W. (2021, January 20\u201325). Pre-trained image processing transformer. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.01212"},{"key":"ref_30","unstructured":"Hu, J., Cao, L., Lu, Y., Zhang, S., Wang, Y., Li, K., Huang, F., Shao, L., and Ji, R. (2021). ISTR: End-to-End Instance Segmentation with Transformers. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TGRS.2020.3034752","article-title":"Remote Sensing Image Change Detection With Transformers","volume":"60","author":"Chen","year":"2021","journal-title":"IEEE Trans. Geosci. Remote. Sens."},{"key":"ref_32","unstructured":"Hendrycks, D., and Gimpel, K. (2016). Gaussian error linear units (gelus). arXiv."},{"key":"ref_33","unstructured":"Ba, J.L., Kiros, J.R., and Hinton, G.E. (2016). Layer normalization. arXiv."},{"key":"ref_34","unstructured":"Contributors, M. (2022, July 17). OpenMMLab\u2019s Image Classification Toolbox and Benchmark. Available online: https:\/\/github.com\/open-mmlab\/mmclassification."},{"key":"ref_35","unstructured":"Contributors, M. (2022, July 17). MMSegmentation: OpenMMLab Semantic Segmentation Toolbox and Benchmark. Available online: https:\/\/github.com\/open-mmlab\/mmsegmentation."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20\u201325). ImageNet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Shi, Q., Li, W., Tao, R., Sun, X., and Gao, L. (2019). Ship classification based on multifeature ensemble with convolutional neural network. Remote Sens., 11.","DOI":"10.3390\/rs11040419"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"13363","DOI":"10.1007\/s11042-017-4952-y","article-title":"Multiple features learning for ship classification in optical imagery","volume":"77","author":"Huang","year":"2018","journal-title":"Multimed. Tools Appl."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Zhang, E., Wang, K., and Lin, G. (2019). Classification of marine vessels with multi-feature structure fusion. Appl. Sci., 9.","DOI":"10.3390\/app9102153"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"3681","DOI":"10.1109\/TGRS.2014.2381602","article-title":"Local binary patterns and extreme learning machine for hyperspectral imagery classification","volume":"53","author":"Li","year":"2015","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Zhao, H., Shi, J., Qi, X., Wang, X., and Jia, J. (2017, January 21\u201326). Pyramid scene parsing network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.660"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., and Lu, H. (2019, January 15\u201320). Dual attention network for scene segmentation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00326"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H. (2018, January 8\u201314). Encoder-decoder with atrous separable convolution for semantic image segmentation. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01234-2_49"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Yuan, Y., Chen, X., and Wang, J. (2020). Object-contextual representations for semantic segmentation. Computer Vision\u2013ECCV 2020, Proceedings of the 16th European Conference, Glasgow, UK, 23\u201328 August 2020, Springer. Part VI 16.","DOI":"10.1007\/978-3-030-58539-6_11"},{"key":"ref_45","unstructured":"Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., and Liu, W. (November, January 27). Ccnet: Criss-cross attention for semantic segmentation. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Seoul, Korea."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/14\/16\/4062\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:12:31Z","timestamp":1760141551000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/14\/16\/4062"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,19]]},"references-count":45,"journal-issue":{"issue":"16","published-online":{"date-parts":[[2022,8]]}},"alternative-id":["rs14164062"],"URL":"https:\/\/doi.org\/10.3390\/rs14164062","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,8,19]]}}}