{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,13]],"date-time":"2026-01-13T15:41:06Z","timestamp":1768318866470,"version":"3.49.0"},"reference-count":38,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2023,1,16]],"date-time":"2023-01-16T00:00:00Z","timestamp":1673827200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"European Space Agency","award":["P2-0103"],"award-info":[{"award-number":["P2-0103"]}]},{"name":"European Space Agency","award":["J2-2505"],"award-info":[{"award-number":["J2-2505"]}]},{"DOI":"10.13039\/501100004329","name":"Slovenian research agency","doi-asserted-by":"publisher","award":["P2-0103"],"award-info":[{"award-number":["P2-0103"]}],"id":[{"id":"10.13039\/501100004329","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004329","name":"Slovenian research agency","doi-asserted-by":"publisher","award":["J2-2505"],"award-info":[{"award-number":["J2-2505"]}],"id":[{"id":"10.13039\/501100004329","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Data in the form of images are now generated at an unprecedented rate. A case in point is remote sensing images (RSI), now available in large-scale RSI archives, which have attracted a considerable amount of research on image classification within the remote sensing community. The basic task of single-target multi-class image classification considers the case where each image is assigned exactly one label from a predefined finite set of class labels. Recently, however, image annotations have become increasingly complex, with images labeled with several labels (instead of just one). In other words, the goal is to assign multiple semantic categories to an image, based on its high-level context. The corresponding machine learning tasks is called multi-label classification (MLC). The classification of RSI is currently predominantly addressed by deep neural network (DNN) approaches, especially convolutional neural networks (CNNs), which can be utilized as feature extractors as well as end-to-end methods. After only considering single-target classification for a long period, DNNs have recently emerged that address the task of MLC. On the other hand, trees and tree ensembles for MLC have a long tradition and are the best-performing class of MLC methods, but need predefined feature representations to operate on. In this work, we explore different strategies for model training based on the transfer learning paradigm, where we utilize different families of (pre-trained) CNN architectures, such as VGG, EfficientNet, and ResNet. The architectures are trained in an end-to-end manner and used in two different modes of operation, namely, as standalone models that directly perform the MLC task, and as feature extractors. In the latter case, the learned representations are used with tree ensemble methods for MLC, such as random forests and extremely randomized trees. We conduct an extensive experimental analysis of methods over several publicly available RSI datasets and evaluate their effectiveness in terms of standard MLC measures. Of these, ranking-based evaluation measures are most relevant, especially ranking loss. The results show that, for addressing the RSI-MLC task, it is favorable to use lightweight network architectures, such as EfficientNet-B2, which is the best performing end-to-end approach, as well as a feature extractor. Furthermore, in the datasets with a limited number of images, using traditional tree ensembles for MLC can yield better performance compared to end-to-end deep approaches.<\/jats:p>","DOI":"10.3390\/rs15020538","type":"journal-article","created":{"date-parts":[[2023,1,17]],"date-time":"2023-01-17T02:58:16Z","timestamp":1673924296000},"page":"538","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":20,"title":["Deep Network Architectures as Feature Extractors for Multi-Label Classification of Remote Sensing Images"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4444-3877","authenticated-orcid":false,"given":"Marjan","family":"Stoimchev","sequence":"first","affiliation":[{"name":"Department of Knowledge Technologies, Jo\u017eef Stefan Institute, Jamova cesta 39, 1000 Ljubljana, Slovenia"},{"name":"Jo\u017eef Stefan International Postgraduate School, 1000 Ljubljana, Slovenia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0687-0878","authenticated-orcid":false,"given":"Dragi","family":"Kocev","sequence":"additional","affiliation":[{"name":"Department of Knowledge Technologies, Jo\u017eef Stefan Institute, Jamova cesta 39, 1000 Ljubljana, Slovenia"},{"name":"Jo\u017eef Stefan International Postgraduate School, 1000 Ljubljana, Slovenia"},{"name":"Bias Variance Labs, 1000 Ljubljana, Slovenia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2363-712X","authenticated-orcid":false,"given":"Sa\u0161o","family":"D\u017eeroski","sequence":"additional","affiliation":[{"name":"Department of Knowledge Technologies, Jo\u017eef Stefan Institute, Jamova cesta 39, 1000 Ljubljana, Slovenia"},{"name":"Jo\u017eef Stefan International Postgraduate School, 1000 Ljubljana, Slovenia"}]}],"member":"1968","published-online":{"date-parts":[[2023,1,16]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"11080","DOI":"10.1109\/JSTARS.2021.3120987","article-title":"Study of Climate Change Detection in North-East Africa Using Machine Learning and Satellite Data","volume":"14","author":"Ibrahim","year":"2021","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_2","first-page":"1","article-title":"Remote Sensing Image Change Detection With Transformers","volume":"60","author":"Chen","year":"2022","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Ortega Adarme, M., Queiroz Feitosa, R., Nigri Happ, P., Aparecido De Almeida, C., and Rodrigues Gomes, A. (2020). Evaluation of Deep Learning Techniques for Deforestation Detection in the Brazilian Amazon and Cerrado Biomes From Remote Sensing Imagery. Remote Sens., 12.","DOI":"10.3390\/rs12060910"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Park, M., Tran, D.Q., Jung, D., and Park, S. (2020). Wildfire-Detection Method Using DenseNet and CycleGAN Data Augmentation-Based Remote Camera Imagery. Remote Sens., 12.","DOI":"10.3390\/rs12223715"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Zhang, Q., Ge, L., Zhang, R., Metternicht, G.I., Liu, C., and Du, Z. (2021). Towards a Deep-Learning-Based Framework of Sentinel-2 Imagery for Automated Active Fire Detection. Remote Sens., 13.","DOI":"10.3390\/rs13234790"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"250","DOI":"10.1016\/j.isprsjprs.2022.11.012","article-title":"Benchmarking and scaling of deep learning models for land cover image classification","volume":"195","author":"Papoutsis","year":"2023","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"4003","DOI":"10.3390\/rs12234003","article-title":"Multi-Label Remote Sensing Image Scene Classification by Combining a Convolutional Neural Network and a Graph Neural Network","volume":"12","author":"Yansheng","year":"2020","journal-title":"Remote Sens."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"117215","DOI":"10.1016\/j.eswa.2022.117215","article-title":"Comprehensive comparative study of multi-label classification methods","volume":"203","author":"Bogatinovski","year":"2022","journal-title":"Expert Syst. Appl."},{"key":"ref_9","unstructured":"Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021, January 3\u20137). An Image is Worth 16\u00d716 Words: Transformers for Image Recognition at Scale. Proceedings of the International Conference on Learning Representations (ICLR), virtual."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L., Kai, L., and Fei-Fei, L. (2009, January 20\u201325). ImageNet: A Large-Scale Hierarchical Image Database. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Dimitrovski, I., Kitanovski, I., Kocev, D., and Simidjievski, N. (2022). Current Trends in Deep Learning for Earth Observation: An Open-source Benchmark Arena for Image Classification. arXiv.","DOI":"10.1016\/j.isprsjprs.2023.01.014"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Pires de Lima, R., and Marfurt, K. (2020). Convolutional Neural Network for Remote-Sensing Scene Classification: Transfer Learning Analysis. Remote Sens., 12.","DOI":"10.3390\/rs12010086"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Khaleghian, S., Ullah, H., Kr\u00e6mer, T., Hughes, N., Eltoft, T., and Marinoni, A. (2021). Sea Ice Classification of SAR Imagery Based on Convolution Neural Networks. Remote Sens., 13.","DOI":"10.3390\/rs13091734"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Wang, A.X., Tran, C., Desai, N., Lobell, D., and Ermon, S. (2018, January 20\u201322). Deep Transfer Learning for Crop Yield Prediction with Remote Sensing Data. Proceedings of the 1st ACM SIGCAS Conference on Computing and Sustainable Societies, San Jose, CA, USA. COMPASS\u201918.","DOI":"10.1145\/3209811.3212707"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Wang, J., Yang, Y., Mao, J., Huang, Z., Huang, C., and Xu, W. (2016, January 27\u201330). CNN-RNN: A Unified Framework for Multi-label Image Classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.251"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Chen, Z., Wei, X., Wang, P., and Guo, Y. (2019, January 15\u201320). Multi-Label Image Recognition with Graph Convolutional Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00532"},{"key":"ref_17","first-page":"5901","article-title":"BigEarthNet: A Large-Scale Benchmark Archive for Remote Sensing Image Understanding","volume":"12","author":"Sumbul","year":"2019","journal-title":"IEEE Int. Geosci. Remote Sens. Symp."},{"key":"ref_18","unstructured":"Yessou, H., Sumbul, G., and Demir, B. (October, January 26). A Comparative Study of Deep Learning Loss Functions for Multi-Label Remote Sensing Image Classification. Proceedings of the IEEE International Geoscience and Remote Sensing Symposium (IGARSS), Waikoloa, HI, USA."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Sumbul, G., Kang, J., and Demir, B. (2020). Deep Learning for Image Search and Retrieval in Large Remote Sensing Archives. arXiv.","DOI":"10.1002\/9781119646181.ch11"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"4558","DOI":"10.1109\/TGRS.2019.2963364","article-title":"Relation Network for Multi-label Aerial Image Classification","volume":"58","author":"Hua","year":"2019","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"95934","DOI":"10.1109\/ACCESS.2020.2995805","article-title":"A Deep Multi-Attention Driven Approach for Multi-Label Remote Sensing Image Classification","volume":"8","author":"Sumbul","year":"2020","journal-title":"IEEE Access"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"11179","DOI":"10.1109\/JSTARS.2021.3122464","article-title":"Global Context-Based Multilevel Feature Fusion Networks for Multilabel Remote Sensing Image Scene Classification","volume":"14","author":"Wang","year":"2021","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_23","unstructured":"Karen, S., and Andrew, Z. (2015, January 7\u20139). Very Deep Convolutional Networks for Large-Scale Image Recognition. Proceedings of the International Conference on Learning Representations (ICLR), San Diego, CA, USA."},{"key":"ref_24","unstructured":"Kaiming, H., Xiangyu, Z., Shaoqing, R., and Jian, S. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA."},{"key":"ref_25","unstructured":"Chaudhuri, K., and Salakhutdinov, R. (2019, January 9\u201315). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, USA."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Yang, Y., and Newsam, S. (2010, January 2\u20135). Bag-of-visual-words and spatial extensions for land-use classification. Proceedings of the 18th ACM SIGSPATIAL International Symposium on Advances in Geographic Information Systems, San Jose, CA, USA.","DOI":"10.1145\/1869790.1869829"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1144","DOI":"10.1109\/TGRS.2017.2760909","article-title":"Multilabel Remote Sensing Image Retrieval Using a Semisupervised Graph-Theoretic Method","volume":"56","author":"Chaudhuri","year":"2018","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"3965","DOI":"10.1109\/TGRS.2017.2685945","article-title":"AID: A Benchmark Data Set for Performance Evaluation of Aerial Scene Classification","volume":"55","author":"Xia","year":"2017","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"\u00d6mr\u00fcuzun, F., Demir, B., L. Bruzzone, L., and \u00c7etin, Y. (2016, January 21\u201324). Content based hyperspectral image retrieval using bag of endmembers image descriptors. Proceedings of the 2016 8th Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS), Los Angeles, CA, USA.","DOI":"10.1109\/WHISPERS.2016.8071805"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"188","DOI":"10.1016\/j.isprsjprs.2019.01.015","article-title":"Recurrently exploring class-wise attention in a hybrid convolutional and bidirectional LSTM network for multi-label aerial image classification","volume":"149","author":"Hua","year":"2019","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"337","DOI":"10.1016\/j.isprsjprs.2020.09.020","article-title":"MLRSNet: A multi-label high spatial resolution remote sensing dataset for semantic scene understanding","volume":"169","author":"Qi","year":"2020","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"174","DOI":"10.1109\/MGRS.2021.3089174","article-title":"BigEarthNet-MM: A Large Scale Multi-Modal Multi-Label Benchmark Archive for Remote Sensing Image Classification and Retrieval","volume":"9","author":"Sumbul","year":"2021","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"817","DOI":"10.1016\/j.patcog.2012.09.023","article-title":"Tree ensembles for predicting structured outputs","volume":"46","author":"Kocev","year":"2013","journal-title":"Pattern Recognit."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"2213","DOI":"10.1007\/s10994-020-05894-4","article-title":"Ensembles of extremely randomized predictive clustering trees for predicting structured outputs","volume":"109","author":"Kocev","year":"2020","journal-title":"Mach. Learn."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Buslaev, A., Iglovikov, V.I., Khvedchenya, E., Parinov, A., Druzhinin, M., and Kalinin, A.A. (2020). Albumentations: Fast and Flexible Image Augmentations. Information, 11.","DOI":"10.3390\/info11020125"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"9176","DOI":"10.1109\/JSTARS.2021.3109600","article-title":"Progressive Data Augmentation Method for Remote Sensing Ship Image Classification Based on Imaging Simulation System and Neural Style Transfer","volume":"14","author":"Xiao","year":"2021","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_37","first-page":"2825","article-title":"Scikit-learn: Machine Learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"ref_38","first-page":"1","article-title":"Statistical Comparisons of Classifiers over Multiple Data Sets","volume":"7","year":"2006","journal-title":"J. Mach. Learn. Res."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/15\/2\/538\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:07:45Z","timestamp":1760119665000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/15\/2\/538"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,16]]},"references-count":38,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2023,1]]}},"alternative-id":["rs15020538"],"URL":"https:\/\/doi.org\/10.3390\/rs15020538","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,16]]}}}