{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,15]],"date-time":"2025-11-15T10:30:11Z","timestamp":1763202611792,"version":"build-2065373602"},"reference-count":31,"publisher":"MDPI AG","issue":"9","license":[{"start":{"date-parts":[[2022,4,28]],"date-time":"2022-04-28T00:00:00Z","timestamp":1651104000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Multimodal hyperspectral and lidar data sets provide complementary spectral and structural data. Joint processing and exploitation to produce semantically labeled pixel maps through semantic segmentation has proven useful for a variety of decision tasks. In this work, we identify two areas of improvement over previous approaches and present a proof of concept network implementing these improvements. First, rather than using a late fusion style architecture as in prior work, our approach implements a composite style fusion architecture to allow for the simultaneous generation of multimodal features and the learning of fused features during encoding. Second, our approach processes the higher information content lidar 3D point cloud data with point-based CNN layers instead of the lower information content lidar 2D DSM used in prior work. Unlike previous approaches, the proof of concept network utilizes a combination of point and pixel-based CNN layers incorporating concatenation-based fusion necessitating a novel point-to-pixel feature discretization method. We characterize our models against a modified GRSS18 data set. Our fusion model achieved 6.6% higher pixel accuracy compared to the highest-performing unimodal model. Furthermore, it achieved 13.5% higher mean accuracy against the hardest to classify samples (14% of total) and equivalent accuracy on the other test set samples.<\/jats:p>","DOI":"10.3390\/rs14092113","type":"journal-article","created":{"date-parts":[[2022,4,28]],"date-time":"2022-04-28T22:20:06Z","timestamp":1651184406000},"page":"2113","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Composite Style Pixel and Point Convolution-Based Deep Fusion Neural Network Architecture for the Semantic Segmentation of Hyperspectral and Lidar Data"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8698-8141","authenticated-orcid":false,"given":"Kevin T.","family":"Decker","sequence":"first","affiliation":[{"name":"Air Force Institute of Technology, Department of Electrical and Computer Engineering, 2950 Hobson Way, Wright Patterson AFB, OH 45433, USA"},{"name":"Air Force Research Laboratory, Multispectral Sensing and Detection Division, LADAR Technology Branch, Wright Patterson AFB, OH 45433, USA"},{"name":"Defense Engineering Corporation (DEC), Beavercreek, OH 45434, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4982-9859","authenticated-orcid":false,"given":"Brett J.","family":"Borghetti","sequence":"additional","affiliation":[{"name":"Air Force Institute of Technology, Department of Electrical and Computer Engineering, 2950 Hobson Way, Wright Patterson AFB, OH 45433, USA"}]}],"member":"1968","published-online":{"date-parts":[[2022,4,28]]},"reference":[{"key":"ref_1","unstructured":"Lillesand, T.M., Kiefer, R.W., and Chipman, J.W. (2015). Remote Sensing and Image Interpretation, John Wiley & Sons, Ltd.. [7th ed.]."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Lahat, D., Adali, T., and Jutten, C. (2015). Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects, IEEE.","DOI":"10.1109\/JPROC.2015.2460697"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"6354","DOI":"10.1109\/TGRS.2017.2726901","article-title":"Fusion of Hyperspectral and LiDAR Data Using Sparse and Low-Rank Component Analysis","volume":"55","author":"Rasti","year":"2017","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"957","DOI":"10.1109\/LGRS.2018.2816958","article-title":"Fusion of Hyperspectral and LiDAR Data with a Novel Ensemble Classifier","volume":"15","author":"Xia","year":"2018","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/j.isprsjprs.2018.06.005","article-title":"Developing a multi-filter convolutional neural network for semantic segmentation using high-resolution aerial imagery and LiDAR data","volume":"143","author":"Sun","year":"2018","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"S40","DOI":"10.1016\/j.isprsjprs.2011.09.012","article-title":"Data fusion of extremely high resolution aerial imagery and LiDAR data for automated railroad centre line reconstruction","volume":"66","author":"Beger","year":"2011","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1369","DOI":"10.1016\/j.rse.2011.01.017","article-title":"Multispectral and LiDAR data fusion for fuel type mapping using Support Vector Machine and decision rules","volume":"115","author":"Chuvieco","year":"2011","journal-title":"Remote Sens. Environ."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1610","DOI":"10.1109\/TGRS.2015.2484325","article-title":"Data fusion technique using wavelet transform and taguchi methods for automatic landslide detection from airborne laser scanning data and quickbird satellite imagery","volume":"54","author":"Pradhan","year":"2016","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Li, C., Tang, X., Shi, L., Peng, Y., and Tang, Y. (2022). A Two-Staged Feature Extraction Method Based on Total Variation for Hyperspectral Images. Remote Sens., 14.","DOI":"10.3390\/rs14020302"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Xu, Y., Du, B., and Zhang, L. (2018, January 22\u201327). Multi-source remote sensing data classification via fully convolutional networks and post-classification processing. Proceedings of the International Geoscience and Remote Sensing Symposium (IGARSS), Valencia, Spain.","DOI":"10.1109\/IGARSS.2018.8518295"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Sukhanov, S., Budylskii, D., Tankoyeu, I., Heremans, R., and Debes, C. (2018, January 22\u201327). Fusion of LiDar, hyperspectral and RGB data for urban land use and land cover classification. Proceedings of the International Geoscience and Remote Sensing Symposium (IGARSS), Valencia, Spain.","DOI":"10.1109\/IGARSS.2018.8517333"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Mohla, S., Pande, S., Banerjee, B., and Chaudhuri, S. (2020, January 14\u201319). FusAtNet: Dual Attention Based SpectroSpatial Multimodal Fusion Network for Hyperspectral and LiDAR Classification. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Seattle, WA, USA.","DOI":"10.21203\/rs.3.rs-32802\/v1"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.inffus.2021.12.008","article-title":"Multi-attentive hierarchical dense fusion net for fusion classification of hyperspectral and LiDAR data","volume":"82","author":"Wang","year":"2022","journal-title":"Inf. Fusion"},{"key":"ref_14","unstructured":"Jetley, S., Lord, N.A., Lee, N., and Torr, P.H. (May, January 30). Learn To Pay Attention. Proceedings of the 6th International Conference on Learning Representations, ICLR 2018\u2014Conference Track Proceedings, Vancouver, BC, Canada."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"110","DOI":"10.1109\/TGRS.2019.2933609","article-title":"Learning to Pay Attention on Spectral Domain: A Spectral Attention Module-Based Convolutional Network for Hyperspectral Image Classification","volume":"58","author":"Mou","year":"2020","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"2405","DOI":"10.1109\/JSTARS.2014.2305441","article-title":"Hyperspectral and LiDAR data fusion: Outcome of the 2013 GRSS data fusion contest","volume":"7","author":"Debes","year":"2014","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Piramanayagam, S., Saber, E., Schwartzkopf, W., and Koehler, F. (2018). Supervised Classification of Multisensor Remotely Sensed Images Using a Deep Learning Framework. Remote Sens., 10.","DOI":"10.3390\/rs10091429"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., and Li, F.F. (2014, January 23\u201328). Large-scale video classification with convolutional neural networks. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.223"},{"key":"ref_19","unstructured":"Guo, Y., Wang, H., Hu, Q., Liu, H., Liu, L., and Bennamoun, M. (2019). Deep Learning for 3D Point Clouds: A Survey. arXiv."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Thomas, H., Qi, C.R., Deschaud, J.E., Marcotegui, B., Goulette, F., and Guibas, L.J. (2019, January 27\u201328). KPConv: Flexible and Deformable Convolution for Point Clouds. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea.","DOI":"10.1109\/ICCV.2019.00651"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1709","DOI":"10.1109\/JSTARS.2019.2911113","article-title":"Advanced multi-sensor optical remote sensing for urban land use and land cover classification: Outcome of the 2018 ieee grss data fusion contest","volume":"12","author":"Xu","year":"2019","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1470","DOI":"10.1109\/LGRS.2019.2944599","article-title":"Learning-Shared Cross-Modality Representation Using Multispectral-LiDAR and Hyperspectral Data","volume":"17","author":"Hong","year":"2020","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Cerra, D., Pato, M., Carmona, E., Azimi, S.M., Tian, J., Bahmanyar, R., Kurz, F., Vig, E., Bittner, K., and Henry, C. (2018, January 22\u201327). Combining deep and shallow neural networks with ad hoc detectors for the classification of complex multi-modal urban scenes. Proceedings of the International Geoscience and Remote Sensing Symposium (IGARSS), Valencia, Spain.","DOI":"10.1109\/IGARSS.2018.8517699"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Fang, S., Quan, D., Wang, S., Zhang, L., and Zhou, L. (2018, January 22\u201327). A two-branch network with semi-supervised learning for hyperspectral classification. Proceedings of the International Geoscience and Remote Sensing Symposium (IGARSS), Valencia, Spain.","DOI":"10.1109\/IGARSS.2018.8517816"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Ronneberger, O., Fischer, P., and Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. arXiv.","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"ref_26","unstructured":"Cao, F., Yang, Z., Ren, J., Jiang, M., and Ling, W.K. (2017). Does Normalization Methods Play a Role for Hyperspectral Image Classification?. arXiv."},{"key":"ref_27","unstructured":"Wallach, H., Larochelle, H., Beygelzimer, A., d\u2019Alch\u00e9-Buc, F., Fox, E., and Garnett, R. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems 32, Curran Associates, Inc."},{"key":"ref_28","unstructured":"Fey, M. (2021, November 15). Pytorch_Scatter. Available online: https:\/\/github.com\/rusty1s\/pytorch_scatter\/releases\/tag\/2.0.9."},{"key":"ref_29","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7\u201312). Going deeper with convolutions. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/14\/9\/2113\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:02:48Z","timestamp":1760137368000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/14\/9\/2113"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,28]]},"references-count":31,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2022,5]]}},"alternative-id":["rs14092113"],"URL":"https:\/\/doi.org\/10.3390\/rs14092113","relation":{},"ISSN":["2072-4292"],"issn-type":[{"type":"electronic","value":"2072-4292"}],"subject":[],"published":{"date-parts":[[2022,4,28]]}}}