{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,3]],"date-time":"2026-04-03T15:18:06Z","timestamp":1775229486272,"version":"3.50.1"},"reference-count":53,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2018,10,16]],"date-time":"2018-10-16T00:00:00Z","timestamp":1539648000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Recently, convolutional neural networks (CNN) have been intensively investigated for the classification of remote sensing data by extracting invariant and abstract features suitable for classification. In this paper, a novel framework is proposed for the fusion of hyperspectral images and LiDAR-derived elevation data based on CNN and composite kernels. First, extinction profiles are applied to both data sources in order to extract spatial and elevation features from hyperspectral and LiDAR-derived data, respectively. Second, a three-stream CNN is designed to extract informative spectral, spatial, and elevation features individually from both available sources. The combination of extinction profiles and CNN features enables us to jointly benefit from low-level and high-level features to improve classification performance. To fuse the heterogeneous spectral, spatial, and elevation features extracted by CNN, instead of a simple stacking strategy, a multi-sensor composite kernels (MCK) scheme is designed. This scheme helps us to achieve higher spectral, spatial, and elevation separability of the extracted features and effectively perform multi-sensor data fusion in kernel space. In this context, a support vector machine and extreme learning machine with their composite kernels version are employed to produce the final classification result. The proposed framework is carried out on two widely used data sets with different characteristics: an urban data set captured over Houston, USA, and a rural data set captured over Trento, Italy. The proposed framework yields the highest OA of     92 . 57 %     and     97 . 91 %     for Houston and Trento data sets. Experimental results confirm that the proposed fusion framework can produce competitive results in both urban and rural areas in terms of classification accuracy, and significantly mitigate the salt and pepper noise in classification maps.<\/jats:p>","DOI":"10.3390\/rs10101649","type":"journal-article","created":{"date-parts":[[2018,10,16]],"date-time":"2018-10-16T11:07:51Z","timestamp":1539688071000},"page":"1649","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":87,"title":["Hyperspectral and LiDAR Fusion Using Deep Three-Stream Convolutional Neural Networks"],"prefix":"10.3390","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6336-8772","authenticated-orcid":false,"given":"Hao","family":"Li","sequence":"first","affiliation":[{"name":"Signal Processing in Earth Observation (SiPEO), Technical University of Munich (TUM), Arcisstr. 21, 80333 Munich, Germany"},{"name":"Institute for Photogrammetry (ifp), University of Stuttgart, 70174 Stuttgart, Germany"},{"name":"GIScience Research Group, Institute of Geography, Heidelberg University, 69120 Heidelberg, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1203-741X","authenticated-orcid":false,"given":"Pedram","family":"Ghamisi","sequence":"additional","affiliation":[{"name":"Helmholtz-Zentrum Dresden-Rossendorf, Helmholtz Institute Freiberg for Resource Technology, Exploration, Chemnitzer Str. 40, D-09599 Freiberg, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4233-6405","authenticated-orcid":false,"given":"Uwe","family":"Soergel","sequence":"additional","affiliation":[{"name":"Institute for Photogrammetry (ifp), University of Stuttgart, 70174 Stuttgart, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5530-3613","authenticated-orcid":false,"given":"Xiao","family":"Zhu","sequence":"additional","affiliation":[{"name":"Signal Processing in Earth Observation (SiPEO), Technical University of Munich (TUM), Arcisstr. 21, 80333 Munich, Germany"},{"name":"Remote Sensing Technology Institute (IMF), German Aerospace Center (DLR), Oberpfaffenhofen, 82234 Wessling, Germany"}]}],"member":"1968","published-online":{"date-parts":[[2018,10,16]]},"reference":[{"key":"ref_1","unstructured":"Benediktsson, J., and Ghamisi, P. (2015). Spectral-Spatial Classification of Hyperspectral Remote Sensing Images, Artech House."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1109\/MGRS.2017.2762087","article-title":"Advances in Hyperspectral Image and Signal Processing: A Comprehensive Overview of the State of the Art","volume":"5","author":"Ghamisi","year":"2017","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"372","DOI":"10.1016\/j.rse.2016.08.018","article-title":"Beyond 3-D: The new spectrum of lidar applications for earth and ecological sciences","volume":"186","author":"Eitel","year":"2016","journal-title":"Remote Sens. Environ."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"134","DOI":"10.1016\/j.isprsjprs.2011.12.003","article-title":"Urban vegetation detection using radiometrically calibrated small-footprint full-waveform airborne LiDAR data","volume":"67","author":"Hollaus","year":"2012","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"319","DOI":"10.1016\/j.inffus.2005.02.007","article-title":"Urban remote sensing using multiple data sets: Past, present, and future","volume":"6","author":"Gamba","year":"2005","journal-title":"Inf. Fusion"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"552","DOI":"10.1109\/LGRS.2014.2350263","article-title":"Generalized graph-based fusion of hyperspectral and LiDAR data using morphological features","volume":"12","author":"Liao","year":"2015","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_7","first-page":"53768","article-title":"Fusion of Hyperspectral and LiDAR data for Classification of Cloud-shadow Mixed Remote Sensing Scene","volume":"10","author":"Luo","year":"2017","journal-title":"IEEE J-STARS"},{"key":"ref_8","first-page":"3011","article-title":"Hyperspectral and LiDAR Data Fusion Using Extinction Profiles and Deep Convolutional Neural Network","volume":"10","author":"Ghamisi","year":"2017","journal-title":"IEEE J-STARS"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"6354","DOI":"10.1109\/TGRS.2017.2726901","article-title":"Fusion of Hyperspectral and LiDAR Data Using Sparse and Low-Rank Component Analysis","volume":"55","author":"Rasti","year":"2017","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1109\/LGRS.2017.2704625","article-title":"Deep Fusion of Remote Sensing Data for Accurate Classification","volume":"14","author":"Chen","year":"2017","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"3997","DOI":"10.1109\/TGRS.2017.2686450","article-title":"Hyperspectral and LiDAR Fusion Using Extinction Profiles and Total Variation Component Analysis","volume":"55","author":"Rasti","year":"2017","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"957","DOI":"10.1080\/2150704X.2017.1335902","article-title":"Classification of hyperspectral and LIDAR data using extinction profiles with feature fusion","volume":"8","author":"Zhang","year":"2017","journal-title":"Remote Sens. Lett."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1080\/19479832.2015.1055833","article-title":"Land-cover classification using both hyperspectral and LiDAR data","volume":"6","author":"Ghamisi","year":"2015","journal-title":"Int. J. Image Data Fusion"},{"key":"ref_14","unstructured":"Ghamisi, P., Cavallaro, G., Wu, D., Benediktsson, J.A., and Plaza, A. (arXiv, 2017). Integration of LiDAR and Hyperspectral Data for Land-cover Classification: A Case Study, arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"309","DOI":"10.1109\/36.905239","article-title":"A new approach for the morphological segmentation of high-resolution satellite imagery","volume":"39","author":"Pesaresi","year":"2005","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"2335","DOI":"10.1109\/TGRS.2014.2358934","article-title":"A Survey on Spectral Spatial Classification Techniques Based on Attribute Profiles","volume":"53","author":"Ghamisi","year":"2015","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"3747","DOI":"10.1109\/TGRS.2010.2048116","article-title":"Morphological Attribute Profiles for the Analysis of Very High Resolution Images","volume":"48","author":"Mura","year":"2010","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1409","DOI":"10.1109\/LGRS.2013.2294241","article-title":"Supervised Segmentation of Very High Resolution Images by the Use of Extended Morphological Attribute Profiles and a Sparse Transform","volume":"11","author":"Li","year":"2014","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"4816","DOI":"10.1109\/TGRS.2012.2230268","article-title":"Genralized Composite Kernel Framework For Hyperspectral Image Classification","volume":"51","author":"Li","year":"2013","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_20","unstructured":"Mura, M.D., Benediktsson, J.A., and Bruzzone, L. (2009). Modeling structural information for building extraction with morphological attribute filters. Image and Signal Processing for Remote Sensing XV, International Society for Optics and Photonics."},{"key":"ref_21","first-page":"2971","article-title":"Fusion of Hyperspectral and LiDAR Remote Sensing Data Using Multiple Feature Learning","volume":"8","author":"Khodadadzadeh","year":"2015","journal-title":"IEEE J-STARS"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"5631","DOI":"10.1109\/TGRS.2016.2561842","article-title":"Extinction Profiles for the Classification of Remote Sensing Data","volume":"54","author":"Ghamisi","year":"2016","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1641","DOI":"10.1109\/LGRS.2016.2600244","article-title":"Hyperspectral Data Classification Using Extended Extinction Profiles","volume":"13","author":"Ghamisi","year":"2016","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"659","DOI":"10.1109\/LGRS.2017.2669304","article-title":"LiDAR Data Classification Using Extinction Profiles and a Composite Kernel Support Vector Machine","volume":"14","author":"Ghamisi","year":"2017","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Long, J., Shelhamer, E., and Darrell, T. (2015, January 7\u201312). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298965"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1109\/MGRS.2017.2762307","article-title":"Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources","volume":"5","author":"Zhu","year":"2017","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_27","first-page":"3435","article-title":"Multitemporal Very High Resolution from Space: Outcome of the 2016 IEEE GRSS Data Fusion Contes","volume":"10","author":"Mou","year":"2017","journal-title":"IEEE J-STARS"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"3639","DOI":"10.1109\/TGRS.2016.2636241","article-title":"Deep Recurrent Neural Networks for Hyperspectral Image Classification","volume":"55","author":"Mou","year":"2017","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Lyu, H., Lu, H., Mou, L., Li, W., Wright, J., Li, X., Li, X., Zhu, X.X., Wang, J., and Yu, L. (2018). Long-Term Annual Mapping of Four Cities on Different Continents by Applying a Deep Information Learning Method to Landsat Data. IEEE J-STARS, 10.","DOI":"10.3390\/rs10030471"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"6232","DOI":"10.1109\/TGRS.2016.2584107","article-title":"Deep Feature Extraction and Classification of Hyperspectral Images Based on Convolutional Neural Networks","volume":"54","author":"Chen","year":"2016","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Mou, L., Bruzzone, L., and Zhu, X.X. (arXiv, 2018). Learning Spectral-Spatial-Temporal Features via a Recurrent Convolutional Neural Network for Change Detection in Multispectral Imagery, arXiv.","DOI":"10.1109\/TGRS.2018.2863224"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1537","DOI":"10.1109\/LGRS.2016.2595108","article-title":"A Self-Improving Convolution Neural Network for the Classification of Hyperspectral Data","volume":"13","author":"Ghamisi","year":"2016","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"784","DOI":"10.1109\/LGRS.2018.2799232","article-title":"Identifying Corresponding Patches in SAR and Optical Images With a Pseudo-Siamese CNN","volume":"15","author":"Hughes","year":"2018","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"5046","DOI":"10.1109\/TGRS.2018.2805286","article-title":"Generative Adversarial Networks for Hyperspectral Image Classification","volume":"56","author":"Zhu","year":"2018","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"391","DOI":"10.1109\/TGRS.2017.2748160","article-title":"Unsupervised Spectral-Spatial Feature Learning via Deep Residual Conv-Deconv Network for Hyperspectral Image Classification","volume":"56","author":"Mou","year":"2018","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_36","first-page":"2351","article-title":"Extreme Learning Machine With Composite Kernels for Hyperspectral Image Classification","volume":"8","author":"Zhou","year":"2015","journal-title":"IEEE J-STARS"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1109\/MGRS.2016.2616418","article-title":"Advanced Spectral Classifiers for Hyperspectral Images: A review","volume":"5","author":"Ghamisi","year":"2017","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_38","unstructured":"Vachier, C. (1995, January 20\u201322). Extinction values: A new measurement of persistence. Proceedings of the 1995 IEEE Workshop on Nonlinear Signal and Image Processing, Halkidiki, Greece."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Soille, P. (2003). Morphological Image Analysis: Principles and Applications, Springer. [2nd ed.].","DOI":"10.1007\/978-3-662-05088-0"},{"key":"ref_40","unstructured":"LeCun, Y., and Bengio, Y. (1998). The Handbook of Brain Theory and Neural Networks, MIT Press."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"Lecun","year":"1998","journal-title":"Proc. IEEE"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Ye, C., Zhao, C., Yang, Y., Ferm\u00fcller, C., and Aloimonos, Y. (arXiv, 2016). LightNet: A Versatile, Standalone Matlab-based Environment for Deep Learning, arXiv.","DOI":"10.1145\/2964284.2973791"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"881","DOI":"10.1109\/TGRS.2016.2616585","article-title":"Dense Semantic Labeling of Subdecimeter Resolution Images With Convolutional Neural Networks","volume":"55","author":"Volpi","year":"2017","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_44","unstructured":"Diamantaras, K., Duch, W., and Iliadis, L.S. (2010). Evaluation of Pooling Operations in Convolutional Architectures for Object Recognition. Artificial Neural Networks\u2013ICANN 2010, Springer."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1109\/TIT.1968.1054102","article-title":"On the mean accuracy of statistical pattern recognizers","volume":"14","author":"Hughes","year":"1968","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"6547","DOI":"10.1109\/TGRS.2017.2729882","article-title":"Multiple Kernel Learning for Hyperspectral Image Classification: A Review","volume":"55","author":"Gu","year":"2017","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"2852","DOI":"10.1109\/TGRS.2011.2176341","article-title":"Representative Multiple Kernel Learning for Classification in Hyperspectral Imagery","volume":"50","author":"Gu","year":"2012","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_48","first-page":"93","article-title":"Composite Kernels for Hyperspectral Image Classification","volume":"3","author":"Munoz","year":"2005","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"1351","DOI":"10.1109\/TGRS.2005.846154","article-title":"Kernel-based methods for hyperspectral image classification","volume":"43","author":"Bruzzone","year":"2005","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"253","DOI":"10.1016\/j.neucom.2013.09.072","article-title":"Multiple kernel extreme learning machine","volume":"149","author":"Liu","year":"2015","journal-title":"Neurocomputing"},{"key":"ref_51","first-page":"2405","article-title":"Hyperspectral and LiDAR Data Fusion: Outcome of the 2013 GRSS Data Fusion Contest","volume":"7","author":"Debes","year":"2014","journal-title":"IEEE J-STARS"},{"key":"ref_52","unstructured":"Nair, V., and Hinton, G.E. (2010, January 21\u201324). Rectified Linear Units Improve Restricted Boltzmann Machines. Proceedings of the 27th International Conference on International Conference on Machine Learning (ICML\u201910), Haifa, Israel."},{"key":"ref_53","unstructured":"Ioffe, S., and Szegedy, C. (arXiv, 2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, arXiv."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/10\/10\/1649\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T15:26:07Z","timestamp":1760196367000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/10\/10\/1649"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,10,16]]},"references-count":53,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2018,10]]}},"alternative-id":["rs10101649"],"URL":"https:\/\/doi.org\/10.3390\/rs10101649","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,10,16]]}}}