{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,17]],"date-time":"2026-06-17T22:36:25Z","timestamp":1781735785638,"version":"3.54.5"},"reference-count":36,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2018,6,16]],"date-time":"2018-06-16T00:00:00Z","timestamp":1529107200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National key R &amp; D plan on strategic international scientific and technological innovation cooperation special project","award":["2016YFE0202300"],"award-info":[{"award-number":["2016YFE0202300"]}]},{"name":"Wuhan Chen Guang Project","award":["2016070204010114"],"award-info":[{"award-number":["2016070204010114"]}]},{"name":"Guangzhou science and technology project","award":["201604020070"],"award-info":[{"award-number":["201604020070"]}]},{"name":"Special task of technical innovation in Hubei Province","award":["2016AAA018"],"award-info":[{"award-number":["2016AAA018"]}]},{"DOI":"10.13039\/501100001809","name":"Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61671332, 41771452 and 41771454"],"award-info":[{"award-number":["61671332, 41771452 and 41771454"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.mdpi.com"],"crossmark-restriction":true},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Benchmark datasets are essential for developing and evaluating remote sensing image retrieval (RSIR) approaches. However, most of the existing datasets are single-labeled, with each image in these datasets being annotated by a single label representing the most significant semantic content of the image. This is sufficient for simple problems, such as distinguishing between a building and a beach, but multiple labels and sometimes even dense (pixel) labels are required for more complex problems, such as RSIR and semantic segmentation.We therefore extended the existing multi-labeled dataset collected for multi-label RSIR and presented a dense labeling remote sensing dataset termed \"DLRSD\". DLRSD contained a total of 17 classes, and the pixels of each image were assigned with 17 pre-defined labels. We used DLRSD to evaluate the performance of RSIR methods ranging from traditional handcrafted feature-based methods to deep learning-based ones. More specifically, we evaluated the performances of RSIR methods from both single-label and multi-label perspectives. These results demonstrated the advantages of multiple labels over single labels for interpreting complex remote sensing images. DLRSD provided the literature a benchmark for RSIR and other pixel-based problems such as semantic segmentation.<\/jats:p>","DOI":"10.3390\/rs10060964","type":"journal-article","created":{"date-parts":[[2018,6,18]],"date-time":"2018-06-18T10:57:11Z","timestamp":1529319431000},"page":"964","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":175,"title":["Performance Evaluation of Single-Label and Multi-Label Remote Sensing Image Retrieval Using a Dense Labeling Dataset"],"prefix":"10.3390","volume":"10","author":[{"given":"Zhenfeng","family":"Shao","sequence":"first","affiliation":[{"name":"State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ke","family":"Yang","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2685-6993","authenticated-orcid":false,"given":"Weixun","family":"Zhou","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2018,6,16]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Bosilj, P., Aptoula, E., Lef\u00e8vre, S., and Kijak, E. (2016). Retrieval of Remote Sensing Images with Pattern Spectra Descriptors. ISPRS Int. J. Geo-Inf., 5.","DOI":"10.3390\/ijgi5120228"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"3023","DOI":"10.1109\/TGRS.2013.2268736","article-title":"Remote sensing image retrieval with global morphological texture descriptors","volume":"52","author":"Aptoula","year":"2014","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Bouteldja, S., and Kourgli, A. (2015, January 10\u201312). Multiscale texture features for the retrieval of high resolution satellite images. Proceedings of the 2015 International Conference on Systems, Signals and Image Processing (IWSSIP), London, UK.","DOI":"10.1109\/IWSSIP.2015.7314204"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"83584","DOI":"10.1117\/1.JRS.8.083584","article-title":"Improved color texture descriptors for remote sensing image retrieval","volume":"8","author":"Shao","year":"2014","journal-title":"J. Appl. Remote Sens."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1603","DOI":"10.1109\/TGRS.2010.2088404","article-title":"Entropy-balanced bitmap tree for shape-based object retrieval from large-scale satellite imagery databases","volume":"49","author":"Scott","year":"2011","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1023\/B:VISI.0000029664.99615.94","article-title":"Distinctive image features from scale-invariant keypoints","volume":"60","author":"Lowe","year":"2004","journal-title":"Int. J. Comput. Vis."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"467","DOI":"10.1109\/TIP.2002.999679","article-title":"Gabor feature based classification using the enhanced fisher linear discriminant model for face recognition","volume":"11","author":"Liu","year":"2002","journal-title":"IEEE Trans. Image Process."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"971","DOI":"10.1109\/TPAMI.2002.1017623","article-title":"Multiresolution gray-scale and rotation invariant texture classification with local binary patterns","volume":"24","author":"Ojala","year":"2002","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"818","DOI":"10.1109\/TGRS.2012.2205158","article-title":"Geographic image retrieval using local invariant features","volume":"51","author":"Yang","year":"2013","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1023\/A:1011139631724","article-title":"Modeling the shape of the scene: A holistic representation of the spatial envelope","volume":"42","author":"Oliva","year":"2001","journal-title":"Int. J. Comput. Vis."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Howarth, P., and R\u00fcger, S. (2004, January 21\u201323). Evaluation of texture features for content-based image retrieval. Proceedings of the International Conference on Image and Video Retrieval, Dublin, Ireland.","DOI":"10.1007\/978-3-540-27814-6_40"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1996","DOI":"10.1109\/LGRS.2014.2316143","article-title":"Performance analysis of state-of-the-art representation methods for geographical image retrieval and categorization","volume":"11","author":"Tola","year":"2014","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Sivic, J., and Zisserman, A. (2003, January 13\u201316). Video Google: A text retrieval approach to object matching in videos. Proceedings of the Ninth IEEE International Conference on Computer Vision, Nice, France.","DOI":"10.1109\/ICCV.2003.1238663"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"J\u00e9gou, H., Douze, M., Schmid, C., and P\u00e9rez, P. (2010, January 13\u201318). Aggregating local descriptors into a compact image representation. Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), San Francisco, CA, USA.","DOI":"10.1109\/CVPR.2010.5540039"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Perronnin, F., S\u00e1nchez, J., and Mensink, T. (2010, January 5\u201311). Improving the Fisher Kernel for Large-Scale Image Classification. Proceedings of the European Conference on Computer Vision, Crete, Greece.","DOI":"10.1007\/978-3-642-15561-1_11"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1080\/17538947.2014.882420","article-title":"An improved Bag-of-Words framework for remote sensing image retrieval in large-scale image databases","volume":"8","author":"Yang","year":"2015","journal-title":"Int. J. Digit. Earth"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Aptoula, E. (2014, January 18\u201320). Bag of morphological words for content-based geographical retrieval. Proceedings of the 2014 12th International Workshop on Content-Based Multimedia Indexing (CBMI), Klagenfurt, Austria.","DOI":"10.1109\/CBMI.2014.6849837"},{"key":"ref_18","unstructured":"Dalal, N., and Triggs, B. (2005, January 20\u201325). Histograms of oriented gradients for human detection. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), San Diego, CA, USA."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Bosch, A., Zisserman, A., and Munoz, X. (2007, January 9\u201311). Representing shape with a spatial pyramid kernel. Proceedings of the 6th ACM International Conference on Image and Video Retrieval, Amsterdam, The Netherlands.","DOI":"10.1145\/1282280.1282340"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"775","DOI":"10.1080\/2150704X.2015.1074756","article-title":"High-resolution remote-sensing imagery retrieval using sparse features by auto-encoder","volume":"6","author":"Zhou","year":"2015","journal-title":"Remote Sens. Lett."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"6020","DOI":"10.1109\/TGRS.2016.2579648","article-title":"A three-layered graph-based learning approach for remote sensing image retrieval","volume":"54","author":"Wang","year":"2016","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Li, F.-F. (2009, January 20\u201325). ImageNet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1343","DOI":"10.1080\/01431161.2017.1399472","article-title":"Visual descriptors for content-based retrieval of remote-sensing images","volume":"39","author":"Napoletano","year":"2018","journal-title":"Int. J. Remote Sens."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Zhou, W., Newsam, S., Li, C., and Shao, Z. (2017). Learning Low Dimensional Convolutional Neural Networks for High-Resolution Remote Sensing Image Retrieval. Remote Sens., 9.","DOI":"10.3390\/rs9050489"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Chatfield, K., Chatfield, K., Simonyan, K., Vedaldi, A., and Zisserman, A. (2014, January 1\u20135). Return of the Devil in the Details: Delving Deep into Convolutional Nets. Proceedings of the British Machine Vision Conference, Nottingham, UK.","DOI":"10.5244\/C.28.6"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Nasierding, G., and Kouzani, A.Z. (2010, January 1\u20133). Empirical study of multi-label classification methods for image annotation and retrieval. Proceedings of the 2010 International Conference on Digital Image Computing: Techniques and Applications (DICTA), Sydney, NSW, Australia.","DOI":"10.1109\/DICTA.2010.113"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Li, R., Zhang, Y., Lu, Z., Lu, J., and Tian, Y. (2010, January 24\u201325). Technique of image retrieval based on multi-label image annotation. Proceedings of the 2010 Second International Conference on Multimedia and Information Technology (MMIT), Kaifeng, China.","DOI":"10.1109\/MMIT.2010.34"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Ranjan, V., Rasiwasia, N., and Jawahar, C.V. (2015, January 7\u201313). Multi-label cross-modal retrieval. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.466"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"2874","DOI":"10.1109\/TGRS.2012.2217397","article-title":"Remote sensing image retrieval by scene semantic matching","volume":"51","author":"Wang","year":"2013","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"4200","DOI":"10.1080\/01431161.2013.774098","article-title":"Remote-sensing image retrieval by combining image visual and semantic features","volume":"34","author":"Wang","year":"2013","journal-title":"Int. J. Remote Sens."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"987","DOI":"10.1109\/LGRS.2016.2558289","article-title":"Region-based retrieval of remote sensing images using an unsupervised graph-theoretic approach","volume":"13","author":"Chaudhuri","year":"2016","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Dai, O.E., Demir, B., Sankur, B., and Bruzzone, L. (2017, January 23\u201328). A novel system for content based retrieval of multi-label remote sensing images. Proceedings of the 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), Fort Worth, TX, USA.","DOI":"10.1109\/IGARSS.2017.8127311"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1144","DOI":"10.1109\/TGRS.2017.2760909","article-title":"Multilabel Remote Sensing Image Retrieval Using a Semisupervised Graph-Theoretic Method","volume":"56","author":"Chaudhuri","year":"2018","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Zhou, W., Newsam, S., Li, C., and Shao, Z. (arXiv, 2017). Patternnet: A benchmark dataset for performance evaluation of remote sensing image retrieval, arXiv.","DOI":"10.1016\/j.isprsjprs.2018.01.004"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Long, J., Shelhamer, E., and Darrell, T. (2015, January 7\u201312). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298965"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. (2014, January 3\u20137). Caffe: Convolutional Architecture for Fast Feature Embedding. Proceedings of the ACM International Conference on Multimedia, Orlando, FL, USA.","DOI":"10.1145\/2647868.2654889"}],"updated-by":[{"DOI":"10.3390\/rs10081220","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2018,6,16]],"date-time":"2018-06-16T00:00:00Z","timestamp":1529107200000}}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/10\/6\/964\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,3]],"date-time":"2025-08-03T22:50:56Z","timestamp":1754261456000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/10\/6\/964"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,6,16]]},"references-count":36,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2018,6]]}},"alternative-id":["rs10060964"],"URL":"https:\/\/doi.org\/10.3390\/rs10060964","relation":{"correction":[{"id-type":"doi","id":"10.3390\/rs10081220","asserted-by":"object"}]},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,6,16]]}}}