{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,7]],"date-time":"2026-01-07T22:55:46Z","timestamp":1767826546686,"version":"3.49.0"},"reference-count":54,"publisher":"MDPI AG","issue":"17","license":[{"start":{"date-parts":[[2020,8,30]],"date-time":"2020-08-30T00:00:00Z","timestamp":1598745600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"DigitAg","award":["ANR-16-CONV-0004"],"award-info":[{"award-number":["ANR-16-CONV-0004"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>European satellite missions Sentinel-1 (S1) and Sentinel-2 (S2) provide at high spatial resolution and high revisit time, respectively, radar and optical images that support a wide range of Earth surface monitoring tasks, such as Land Use\/Land Cover mapping. A long-standing challenge in the remote sensing community is about how to efficiently exploit multiple sources of information and leverage their complementarity, in order to obtain the most out of radar and optical data. In this work, we propose to deal with land cover mapping in an object-based image analysis (OBIA) setting via a deep learning framework designed to leverage the multi-source complementarity provided by radar and optical satellite image time series (SITS). The proposed architecture is based on an extension of Recurrent Neural Network (RNN) enriched via a modified attention mechanism capable to fit the specificity of SITS data. Our framework also integrates a pretraining strategy that allows to exploit specific domain knowledge, shaped as hierarchy over the set of land cover classes, to guide the model training. Thorough experimental evaluations, involving several competitive approaches were conducted on two study sites, namely the Reunion island and a part of the Senegalese groundnut basin. Classification results, 79% of global accuracy on the Reunion island and 90% on the Senegalese site, respectively, have demonstrated the suitability of the proposal.<\/jats:p>","DOI":"10.3390\/rs12172814","type":"journal-article","created":{"date-parts":[[2020,8,30]],"date-time":"2020-08-30T22:00:17Z","timestamp":1598824817000},"page":"2814","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":31,"title":["Object-Based Multi-Temporal and Multi-Source Land Cover Mapping Leveraging Hierarchical Class Relationships"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6923-0248","authenticated-orcid":false,"given":"Yawogan Jean Eudes","family":"Gbodjo","sequence":"first","affiliation":[{"name":"INRAE, UMR TETIS, University of Montpellier, 34000 Montpellier, France"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8736-3132","authenticated-orcid":false,"given":"Dino","family":"Ienco","sequence":"additional","affiliation":[{"name":"INRAE, UMR TETIS, University of Montpellier, 34000 Montpellier, France"},{"name":"LIRMM, University of Montpellier, 34095 Montpellier, France"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7631-2399","authenticated-orcid":false,"given":"Louise","family":"Leroux","sequence":"additional","affiliation":[{"name":"Cirad, UPR A\u00cfDA, Dakar, S\u00e9n\u00e9gal and A\u00cfDA, University of Montpellier, Cirad, 34398 Montpellier, France"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0536-6277","authenticated-orcid":false,"given":"Roberto","family":"Interdonato","sequence":"additional","affiliation":[{"name":"Cirad, UMR TETIS, 34000 Montpellier, France"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9470-4791","authenticated-orcid":false,"given":"Raffaele","family":"Gaetano","sequence":"additional","affiliation":[{"name":"Cirad, UMR TETIS, 34000 Montpellier, France"}]},{"given":"Babacar","family":"Ndao","sequence":"additional","affiliation":[{"name":"Centre de Suivi Ecologique and University Cheikh Anta Diop, 15532 Dakar, Senegal"}]}],"member":"1968","published-online":{"date-parts":[[2020,8,30]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1016\/j.rse.2011.07.023","article-title":"ESA\u2019s sentinel missions in support of Earth system science","volume":"120","author":"Berger","year":"2012","journal-title":"Remote Sens. Environ."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Kolecka, N., Ginzler, C., Pazur, R., Price, B., and Verburg, P.H. (2018). Regional Scale Mapping of Grassland Mowing Frequency with Sentinel-2 Time Series. Remote Sens., 10.","DOI":"10.3390\/rs10081221"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"778","DOI":"10.1109\/LGRS.2017.2681128","article-title":"Deep Learning Classification of Land Cover and Crop Types Using Remote Sensing Data","volume":"14","author":"Kussul","year":"2017","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Inglada, J., Vincent, A., Arias, M., Tardy, B., Morin, D., and Rodes, I. (2017). Operational High Resolution Land Cover Map Production at the Country Scale Using Satellite Image Time Series. Remote Sens., 9.","DOI":"10.3390\/rs9010095"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1016\/j.isprsjprs.2017.05.013","article-title":"A graph-based approach to detect spatiotemporal dynamics in satellite image time series","volume":"130","author":"Guttler","year":"2017","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1016\/j.ecoinf.2017.11.003","article-title":"Object-oriented satellite image time series analysis using a graph-based representation","volume":"43","author":"Khiali","year":"2018","journal-title":"Ecol. Inform."},{"key":"ref_7","first-page":"595","article-title":"Combining Sentinel-1 and Sentinel-2 data for improved land use and land cover mapping of monsoon regions","volume":"73","author":"Steinhausen","year":"2018","journal-title":"Int. J. Appl. Earth Obs. Geoinf."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"464","DOI":"10.1109\/LGRS.2018.2794581","article-title":"Deep Recurrent Neural Networks for Winter Vegetation Quality Mapping via Multitemporal SAR Sentinel-1","volume":"15","author":"Minh","year":"2018","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"307","DOI":"10.1109\/LGRS.2019.2917788","article-title":"Toward Spatio-Spectral Analysis of Sentinel-2 Time Series Data for Land Cover Mapping","volume":"17","author":"Gbodjo","year":"2019","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1016\/j.isprsjprs.2019.01.011","article-title":"DuPLO: A DUal view Point deep Learning architecture for time series classificatiOn","volume":"149","author":"Interdonato","year":"2019","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1155","DOI":"10.1080\/19475705.2018.1549112","article-title":"Estimation of flood land use\/land cover mapping by regional modelling of flood hazard at sub-basin level case study: Marand basin","volume":"10","author":"Mousavi","year":"2019","journal-title":"Geomat. Nat. Hazards Risk"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"258","DOI":"10.1016\/j.agsy.2018.05.010","article-title":"A comparison of global agricultural monitoring systems and current gaps","volume":"168","author":"Fritz","year":"2019","journal-title":"Agric. Syst."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"2207","DOI":"10.1109\/TGRS.2006.872081","article-title":"On the blending of the Landsat and MODIS surface reflectance: Predicting daily Landsat surface reflectance","volume":"44","author":"Gao","year":"2006","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1016\/j.isprsjprs.2019.09.016","article-title":"Combining Sentinel-1 and Sentinel-2 Satellite Image Time Series for land cover mapping via a multi-source deep learning architecture","volume":"158","author":"Ienco","year":"2019","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Ienco, D., Gaetano, R., Interdonato, R., Ose, K., and Minh, D.H.T. (August, January 28). Combining Sentinel-1 and Sentinel-2 Time Series via RNN for Object-Based Land Cover Classification. Proceedings of the 2019 IEEE International Geoscience and Remote Sensing Symposium (IGARSS 2019), Yokohama, Japan.","DOI":"10.1109\/IGARSS.2019.8898458"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Iannelli, G.C., and Gamba, P. (2018, January 22\u201327). Jointly Exploiting Sentinel-1 and Sentinel-2 for Urban Mapping. Proceedings of the 2018 IEEE International Geoscience and Remote Sensing Symposium (IGARSS 2018), Valencia, Spain.","DOI":"10.1109\/IGARSS.2018.8518172"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"345","DOI":"10.1016\/j.rse.2018.07.006","article-title":"Mapping and assessment of vegetation types in the tropical rainforests of the Western Ghats using multispectral Sentinel-2 and SAR Sentinel-1 satellite imagery","volume":"216","author":"Erinjery","year":"2018","journal-title":"Remote Sens. Environ."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Tricht, K.V., Gobin, A., Gilliams, S., and Piccard, I. (2018). Synergistic Use of Radar Sentinel-1 and Optical Sentinel-2 Imagery for Crop Mapping: A Case Study for Belgium. Remote Sens., 10.","DOI":"10.3390\/rs10101642"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Denize, J., Hubert-Moy, L., Betbeder, J., Corgne, S., Baudry, J., and Pottier, E. (2019). Evaluation of using sentinel-1 and-2 time-series to identify winter land use in agricultural landscapes. Remote Sens., 11.","DOI":"10.3390\/rs11010037"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1347","DOI":"10.1109\/LGRS.2018.2843886","article-title":"Multimodal Probabilistic Latent Semantic Analysis for Sentinel-1 and Sentinel-2 Image Fusion","volume":"15","author":"Haut","year":"2018","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_21","unstructured":"Di Gregorio, A. (2005). Land Cover Classification System: Classification Concepts and User Manual: LCCS, Food & Agriculture Organization."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"392","DOI":"10.1016\/j.rse.2010.09.010","article-title":"Hierarchical mapping of Northern Eurasian land cover using MODIS data","volume":"115","author":"Friedl","year":"2011","journal-title":"Remote Sens. Environ."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"012016","DOI":"10.1088\/1755-1315\/46\/1\/012016","article-title":"A Hierarchical Object-oriented Urban Land Cover Classification Using WorldView-2 Imagery and Airborne LiDAR data","volume":"46","author":"Wu","year":"2016","journal-title":"IOP Conf. Ser. Earth Environ. Sci."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1016\/j.rse.2018.12.013","article-title":"Hierarchical mapping of annual global land cover 2001 to present: The MODIS Collection 6 Land Cover product","volume":"222","author":"Gray","year":"2019","journal-title":"Remote Sens. Environ."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1016\/j.isprsjprs.2009.06.004","article-title":"Object based image analysis for remote sensing","volume":"65","author":"Blaschke","year":"2010","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_26","unstructured":"Lillesand, T., Kiefer, R.W., and Chipman, J. (2015). Remote Sensing and Image Interpretation, John Wiley & Sons."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1109\/MGRS.2017.2762307","article-title":"Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources","volume":"5","author":"Zhu","year":"2017","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1685","DOI":"10.1109\/LGRS.2017.2728698","article-title":"Land Cover Classification via Multitemporal Spatial Data by Deep Recurrent Neural Networks","volume":"14","author":"Ienco","year":"2017","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"391","DOI":"10.1109\/TGRS.2017.2748160","article-title":"Unsupervised Spectral-Spatial Feature Learning via Deep Residual Conv-Deconv Network for Hyperspectral Image Classification","volume":"56","author":"Mou","year":"2018","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"430","DOI":"10.1016\/j.rse.2018.11.032","article-title":"Deep learning based multi-temporal crop classification","volume":"221","author":"Zhong","year":"2019","journal-title":"Remote Sens. Environ."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Ru\u00dfwurm, M., and K\u00f6rner, M. (2018). Multi-Temporal Land Cover Classification with Sequential Recurrent Encoders. ISPRS Int. J. Geo-Inf., 7.","DOI":"10.3390\/ijgi7040129"},{"key":"ref_32","unstructured":"Rouse, J.W., Hass, R.H., Schell, J., and Deering, D. (1973, January 10\u201314). Monitoring vegetation systems in the great plains with ERTS. Proceedings of the Third Earth Resources Technology Satellite (ERTS) symposium, Washington, DC, USA."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"104934","DOI":"10.1016\/j.dib.2019.104934","article-title":"Mapping land cover on Reunion Island in 2017 using satellite imagery and geospatial ground data","volume":"28","author":"Dupuy","year":"2020","journal-title":"Data Brief"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"5473","DOI":"10.1109\/TGRS.2015.2422848","article-title":"A Scalable Tile-Based Framework for Region-Merging Segmentation","volume":"53","author":"Lassalle","year":"2015","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Cho, K., Van Merrienboer, B., G\u00fcl\u00e7ehre, \u00c7., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. arXiv.","DOI":"10.3115\/v1\/D14-1179"},{"key":"ref_36","unstructured":"Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural Machine Translation by Jointly Learning to Align and Translate. arXiv."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Luong, M., Pham, H., and Manning, C.D. (2015, January 17\u201321). Effective Approaches to Attention-based Neural Machine Translation. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015), Lisbon, Portugal.","DOI":"10.18653\/v1\/D15-1166"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Britz, D., Guan, M.Y., and Luong, M. (2017). Efficient Attention using a Fixed-Size Memory Representation. arXiv.","DOI":"10.18653\/v1\/D17-1040"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Karamanolakis, G., Hsu, D., and Gravano, L. (2019). Weakly Supervised Attention Networks for Fine-Grained Opinion Mining and Public Health. arXiv.","DOI":"10.18653\/v1\/D19-5501"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Hou, S., Liu, X., and Wang, Z. (2017, January 22\u201329). DualNet: Learn Complementary Features for Image Recognition. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.62"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"4939","DOI":"10.1109\/JSTARS.2018.2876357","article-title":"M3 Fusion: A Deep Learning Architecture for Multiscale Multimodal Multitemporal Satellite Data Fusion","volume":"11","author":"Benedetti","year":"2018","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Valero, S., Arnaud, L., Planells, M., Ceschia, E., and Dedieu, G. (August, January 28). Sentinel\u2019s Classifier Fusion System for Seasonal Crop Mapping. Proceedings of the 2019 IEEE International Geoscience and Remote Sensing Symposium (IGARSS 2019), Yokohama, Japan.","DOI":"10.1109\/IGARSS.2019.8898011"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Pelletier, C., Webb, G., and Petitjean, F. (2019). Temporal Convolutional Neural Network for the Classification of Satellite Image Time Series. Remote Sens., 11.","DOI":"10.3390\/rs11050523"},{"key":"ref_44","first-page":"2825","article-title":"Scikit-learn: Machine Learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"ref_45","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"Lecun","year":"2015","journal-title":"Nature"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"166","DOI":"10.1016\/j.isprsjprs.2019.04.015","article-title":"Deep learning in remote sensing applications: A meta-analysis and review","volume":"152","author":"Ma","year":"2019","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"2784","DOI":"10.1080\/01431161.2018.1433343","article-title":"Implementation of machine-learning classification in remote sensing: An applied review","volume":"39","author":"Maxwell","year":"2018","journal-title":"Int. J. Remote Sens."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1016\/j.neucom.2018.01.007","article-title":"Fine-grained attention mechanism for neural machine translation","volume":"284","author":"Choi","year":"2018","journal-title":"Neurocomputing"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 13\u201317). \u201cWhy Should I Trust You?\u201d: Explaining the Predictions of Any Classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.","DOI":"10.1145\/2939672.2939778"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1080\/15481603.2018.1426092","article-title":"Geographic object-based image analysis (GEOBIA): Emerging trends and future opportunities","volume":"55","author":"Chen","year":"2018","journal-title":"GISci. Remote Sens."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Boccardo, P., and Tonolo, F.G. (2015). Remote sensing role in emergency mapping for disaster response. Engineering Geology for Society and Territory\u2014Volume 5, Springer.","DOI":"10.1007\/978-3-319-09048-1_3"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1016\/j.isprsjprs.2016.01.011","article-title":"Random forest in remote sensing: A review of applications and future directions","volume":"114","author":"Belgiu","year":"2016","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_54","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4\u20139). Attention is All you Need. Proceedings of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, Long Beach, CA, USA."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/12\/17\/2814\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:04:52Z","timestamp":1760177092000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/12\/17\/2814"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,30]]},"references-count":54,"journal-issue":{"issue":"17","published-online":{"date-parts":[[2020,9]]}},"alternative-id":["rs12172814"],"URL":"https:\/\/doi.org\/10.3390\/rs12172814","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,8,30]]}}}