{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T17:41:42Z","timestamp":1777657302475,"version":"3.51.4"},"reference-count":39,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2020,6,26]],"date-time":"2020-06-26T00:00:00Z","timestamp":1593129600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Multi-sensor remote sensing image classification has been considerably improved by deep learning feature extraction and classification networks. In this paper, we propose a novel multi-sensor fusion framework for the fusion of diverse remote sensing data sources. The novelty of this paper is grounded in three important design innovations: 1- a unique adaptation of the coupled residual networks to address multi-sensor data classification; 2- a smart auxiliary training via adjusting the loss function to address classifications with limited samples; and 3- a unique design of the residual blocks to reduce the computational complexity while preserving the discriminative characteristics of multi-sensor features. The proposed classification framework is evaluated using three different remote sensing datasets: the urban Houston university datasets (including Houston 2013 and the training portion of Houston 2018) and the rural Trento dataset. The proposed framework achieves high overall accuracies of 93.57%, 81.20%, and 98.81% on Houston 2013, the training portion of Houston 2018, and Trento datasets, respectively. Additionally, the experimental results demonstrate considerable improvements in classification accuracies compared with the existing state-of-the-art methods.<\/jats:p>","DOI":"10.3390\/rs12122067","type":"journal-article","created":{"date-parts":[[2020,6,29]],"date-time":"2020-06-29T11:17:17Z","timestamp":1593429437000},"page":"2067","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":26,"title":["A Multi-Sensor Fusion Framework Based on Coupled Residual Convolutional Neural Networks"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6336-8772","authenticated-orcid":false,"given":"Hao","family":"Li","sequence":"first","affiliation":[{"name":"GIScience Chair, Institute of Geography, Heidelberg University, 69120 Heidelberg, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1203-741X","authenticated-orcid":false,"given":"Pedram","family":"Ghamisi","sequence":"additional","affiliation":[{"name":"Helmholtz-Zentrum Dresden-Rossendorf, Helmholtz Institute Freiberg for Resource Technology, Exploration, Chemnitzer Str. 40, D-09599 Freiberg, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Behnood","family":"Rasti","sequence":"additional","affiliation":[{"name":"Helmholtz-Zentrum Dresden-Rossendorf, Helmholtz Institute Freiberg for Resource Technology, Exploration, Chemnitzer Str. 40, D-09599 Freiberg, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5639-0317","authenticated-orcid":false,"given":"Zhaoyan","family":"Wu","sequence":"additional","affiliation":[{"name":"GIScience Chair, Institute of Geography, Heidelberg University, 69120 Heidelberg, Germany"},{"name":"The School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Aurelie","family":"Shapiro","sequence":"additional","affiliation":[{"name":"Here+There Mapping Solutions, 10115 Berlin, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael","family":"Schultz","sequence":"additional","affiliation":[{"name":"GIScience Chair, Institute of Geography, Heidelberg University, 69120 Heidelberg, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4916-9838","authenticated-orcid":false,"given":"Alexander","family":"Zipf","sequence":"additional","affiliation":[{"name":"GIScience Chair, Institute of Geography, Heidelberg University, 69120 Heidelberg, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,6,26]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1109\/MGRS.2018.2890023","article-title":"Multisource and Multitemporal Data Fusion in Remote Sensing: A Comprehensive Review of the State of the Art","volume":"7","author":"Ghamisi","year":"2019","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1147","DOI":"10.1126\/science.228.4704.1147","article-title":"Imaging Spectrometry for Earth Remote Sensing","volume":"228","author":"Goetz","year":"1985","journal-title":"Science"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1109\/MGRS.2013.2244672","article-title":"Hyperspectral Remote Sensing Data Analysis and Future Challenges","volume":"1","author":"Plaza","year":"2013","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1109\/MGRS.2017.2762087","article-title":"Advances in Hyperspectral Image and Signal Processing: A Comprehensive Overview of the State of the Art","volume":"5","author":"Ghamisi","year":"2017","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"372","DOI":"10.1016\/j.rse.2016.08.018","article-title":"Beyond 3-D: The new spectrum of lidar applications for earth and ecological sciences","volume":"186","author":"Eitel","year":"2016","journal-title":"Remote. Sens. Environ."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"134","DOI":"10.1016\/j.isprsjprs.2011.12.003","article-title":"Urban vegetation detection using radiometrically calibrated small-footprint full-waveform airborne LiDAR data","volume":"67","author":"Hollaus","year":"2012","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"352","DOI":"10.1016\/j.isprsjprs.2019.11.025","article-title":"4D objects-by-change: Spatiotemporal segmentation of geomorphic surface change from LiDAR time series","volume":"159","author":"Anders","year":"2020","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"3011","DOI":"10.1109\/JSTARS.2016.2634863","article-title":"Hyperspectral and LiDAR Data Fusion Using Extinction Profiles and Deep Convolutional Neural Network","volume":"10","author":"Ghamisi","year":"2017","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_9","unstructured":"H\u00e4nsch, R., and Hellwich, O. (2020). Fusion of Multispectral LiDAR, Hyperspectral, and RGB Data for Urban Land Cover Classification. IEEE Geosci. Remote. Sens. Lett., 1\u20135."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"2971","DOI":"10.1109\/JSTARS.2015.2432037","article-title":"Fusion of Hyperspectral and LiDAR Remote Sensing Data Using Multiple Feature Learning","volume":"8","author":"Khodadadzadeh","year":"2015","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"856","DOI":"10.1109\/JSTSP.2012.2208177","article-title":"Classification of Remote Sensing Optical and LiDAR Data Using Extended Attribute Profiles","volume":"6","author":"Pedergnana","year":"2012","journal-title":"IEEE J. Sel. Top. Signal Process."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"5631","DOI":"10.1109\/TGRS.2016.2561842","article-title":"Extinction Profiles for the Classification of Remote Sensing Data","volume":"54","author":"Ghamisi","year":"2016","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"3997","DOI":"10.1109\/TGRS.2017.2686450","article-title":"Hyperspectral and LiDAR Fusion Using Extinction Profiles and Total Variation Component Analysis","volume":"55","author":"Rasti","year":"2017","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"552","DOI":"10.1109\/LGRS.2014.2350263","article-title":"Generalized graph-based fusion of hyperspectral and LiDAR data using morphological features","volume":"12","author":"Liao","year":"2015","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"6354","DOI":"10.1109\/TGRS.2017.2726901","article-title":"Fusion of Hyperspectral and LiDAR Data Using Sparse and Low-Rank Component Analysis","volume":"55","author":"Rasti","year":"2017","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"196","DOI":"10.1109\/LGRS.2018.2869888","article-title":"Multisensor Composite Kernels Based on Extreme Learning Machines","volume":"16","author":"Ghamisi","year":"2019","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Zhong, Y., Cao, Q., Zhao, J., Ma, A., Zhao, B., and Zhang, L. (2017). Optimal Decision Fusion for Urban Land-Use\/Land-Cover Classification Based on Adaptive Differential Evolution Using Hyperspectral and LiDAR Data. Remote Sens., 9.","DOI":"10.3390\/rs9080868"},{"key":"ref_18","unstructured":"Xia, J., Liao, W., and Du, P. (2019). Hyperspectral and LiDAR Classification With Semisupervised Graph Fusion. IEEE Geosci. Remote Sens. Lett., 1\u20135."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"3905","DOI":"10.1109\/JSTARS.2018.2868142","article-title":"Fusion of Hyperspectral and LiDAR Data Using Discriminant Correlation Analysis for Land Cover Classification","volume":"11","author":"Jahan","year":"2018","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1109\/TIT.1968.1054102","article-title":"On the mean accuracy of statistical pattern recognizers","volume":"14","author":"Hughes","year":"1968","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"6690","DOI":"10.1109\/TGRS.2019.2907932","article-title":"Deep Learning for Hyperspectral Image Classification: An Overview","volume":"57","author":"Li","year":"2019","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Rasti, B., Hong, D., Hang, R., Ghamisi, P., Kang, X., Chanussot, J., and Benediktsson, J.A. (2020). Feature Extraction for Hyperspectral Imagery: The Evolution from Shallow to Deep (Overview and Toolbox). IEEE Geosci. Remote Sens. Mag.","DOI":"10.1109\/MGRS.2020.2979764"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1253","DOI":"10.1109\/LGRS.2017.2704625","article-title":"Deep Fusion of Remote Sensing Data for Accurate Classification","volume":"14","author":"Chen","year":"2017","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Li, H., Ghamisi, P., Soergel, U., and Zhu, X.X. (2018). Hyperspectral and LiDAR Fusion Using Deep Three-Stream Convolutional Neural Networks. Remote Sens., 10.","DOI":"10.3390\/rs10101649"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"937","DOI":"10.1109\/TGRS.2017.2756851","article-title":"Multisource Remote Sensing Data Classification Based on Convolutional Neural Network","volume":"56","author":"Xu","year":"2018","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Qiu, C., Schmitt, M., Mou, L., Ghamisi, P., and Zhu, X.X. (2018). Feature Importance Analysis for Local Climate Zone Classification Using a Residual Convolutional Neural Network with Multi-Source Datasets. Remote Sens., 10.","DOI":"10.3390\/rs10101572"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1109\/TCYB.2018.2864670","article-title":"Feature Extraction for Classification of Hyperspectral and LiDAR Data Using Patch-to-Patch CNN","volume":"50","author":"Zhang","year":"2020","journal-title":"IEEE Trans. Cybern."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Xu, S., Amira, O., Liu, J., Zhang, C., Zhang, J., and Li, G. (2020). HAM-MFN: Hyperspectral and Multispectral Image Multiscale Fusion Network With RAP Loss. IEEE Trans. Geosci. Remote Sens., 1\u201311.","DOI":"10.1109\/TGRS.2020.2964777"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Leibe, B., Matas, J., Sebe, N., and Welling, M. (2016). Identity Mappings in Deep Residual Networks. Computer Vision \u2013 ECCV 2016, Springer International Publishing.","DOI":"10.1007\/978-3-319-46454-1"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"847","DOI":"10.1109\/TGRS.2017.2755542","article-title":"Spectral\u2013Spatial Residual Network for Hyperspectral Image Classification: A 3-D Deep Learning Framework","volume":"56","author":"Zhong","year":"2018","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Qiu, C., Mou, L., Schmitt, M., and Zhu, X.X. (2020). Fusing Multiseasonal Sentinel-2 Imagery for Urban Land Cover Classification With Multibranch Residual Convolutional Neural Networks. IEEE Geosci. Remote Sens. Lett., 1\u20135.","DOI":"10.1109\/LGRS.2019.2953497"},{"key":"ref_33","first-page":"448","article-title":"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift","volume":"Volume 37","author":"Bach","year":"2015","journal-title":"Proceedings of the 32nd International Conference on Machine Learning"},{"key":"ref_34","unstructured":"Nair, V., and Hinton, G.E. (2010, January 21\u201324). Rectified Linear Units Improve Restricted Boltzmann Machines. Proceedings of the 27th International Conference on Machine Learning (ICML 2010), Haifa, Israel."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"6232","DOI":"10.1109\/TGRS.2016.2584107","article-title":"Deep Feature Extraction and Classification of Hyperspectral Images Based on Convolutional Neural Networks","volume":"54","author":"Chen","year":"2016","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1641","DOI":"10.1109\/LGRS.2016.2600244","article-title":"Hyperspectral Data Classification Using Extended Extinction Profiles","volume":"13","author":"Ghamisi","year":"2016","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7\u201312). Going deeper with convolutions. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"2405","DOI":"10.1109\/JSTARS.2014.2305441","article-title":"Hyperspectral and LiDAR Data Fusion: Outcome of the 2013 GRSS Data Fusion Contest","volume":"7","author":"Debes","year":"2014","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1709","DOI":"10.1109\/JSTARS.2019.2911113","article-title":"Advanced Multi-Sensor Optical Remote Sensing for Urban Land Use and Land Cover Classification: Outcome of the 2018 IEEE GRSS Data Fusion Contest","volume":"12","author":"Xu","year":"2019","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/12\/12\/2067\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:43:30Z","timestamp":1760175810000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/12\/12\/2067"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,26]]},"references-count":39,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2020,6]]}},"alternative-id":["rs12122067"],"URL":"https:\/\/doi.org\/10.3390\/rs12122067","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,6,26]]}}}