{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,10]],"date-time":"2026-03-10T09:19:19Z","timestamp":1773134359838,"version":"3.50.1"},"reference-count":50,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2023,1,31]],"date-time":"2023-01-31T00:00:00Z","timestamp":1675123200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"\u201cCentral Innovation Programme for small and medium-sized enterprises (ZIM)\u201d of the German Federal Ministry for Economic Affairs and Climate Action","award":["16KN089024"],"award-info":[{"award-number":["16KN089024"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>In this article, we propose a deep learning-based algorithm for the classification of crop types from Sentinel-1 and Sentinel-2 time series data which is based on the celebrated transformer architecture. Crucially, we enable our algorithm to do early classification, i.e., predict crop types at arbitrary time points early in the year with a single trained model (progressive intra-season classification). Such early season predictions are of practical relevance for instance for yield forecasts or the modeling of agricultural water balances, therefore being important for the public as well as the private sector. Furthermore, we improve the mechanism of combining different data sources for the prediction task, allowing for both optical and radar data as inputs (multi-modal data fusion) without the need for temporal interpolation. We can demonstrate the effectiveness of our approach on an extensive data set from three federal states of Germany reaching an average F1 score of 0.92 using data of a complete growing season to predict the eight most important crop types and an F1 score above 0.8 when doing early classification at least one month before harvest time. In carefully chosen experiments, we can show that our model generalizes well in time and space.<\/jats:p>","DOI":"10.3390\/rs15030799","type":"journal-article","created":{"date-parts":[[2023,2,1]],"date-time":"2023-02-01T05:33:53Z","timestamp":1675229633000},"page":"799","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":40,"title":["Early Crop Classification via Multi-Modal Satellite Data Fusion and Temporal Attention"],"prefix":"10.3390","volume":"15","author":[{"given":"Frank","family":"Weilandt","sequence":"first","affiliation":[{"name":"dida Datenschmiede GmbH, Hauptstr. 8, Meisenbach H\u00f6fe, 10827 Berlin, Germany"}]},{"given":"Robert","family":"Behling","sequence":"additional","affiliation":[{"name":"Helmholtz-Zentrum Potsdam, Deutsches GeoForschungsZentrum GFZ Telegrafenberg, 14473 Potsdam, Germany"}]},{"given":"Romulo","family":"Goncalves","sequence":"additional","affiliation":[{"name":"Helmholtz-Zentrum Potsdam, Deutsches GeoForschungsZentrum GFZ Telegrafenberg, 14473 Potsdam, Germany"}]},{"given":"Arash","family":"Madadi","sequence":"additional","affiliation":[{"name":"Helmholtz-Zentrum Potsdam, Deutsches GeoForschungsZentrum GFZ Telegrafenberg, 14473 Potsdam, Germany"}]},{"given":"Lorenz","family":"Richter","sequence":"additional","affiliation":[{"name":"dida Datenschmiede GmbH, Hauptstr. 8, Meisenbach H\u00f6fe, 10827 Berlin, Germany"},{"name":"Zuse Institute Berlin, Takustra\u00dfe 7, 14195 Berlin, Germany"}]},{"given":"Tiago","family":"Sanona","sequence":"additional","affiliation":[{"name":"dida Datenschmiede GmbH, Hauptstr. 8, Meisenbach H\u00f6fe, 10827 Berlin, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2939-8764","authenticated-orcid":false,"given":"Daniel","family":"Spengler","sequence":"additional","affiliation":[{"name":"Helmholtz-Zentrum Potsdam, Deutsches GeoForschungsZentrum GFZ Telegrafenberg, 14473 Potsdam, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7165-1589","authenticated-orcid":false,"given":"Jona","family":"Welsch","sequence":"additional","affiliation":[{"name":"dida Datenschmiede GmbH, Hauptstr. 8, Meisenbach H\u00f6fe, 10827 Berlin, Germany"}]}],"member":"1968","published-online":{"date-parts":[[2023,1,31]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Mahlayeye, M., Darvishzadeh, R., and Nelson, A. (2022). Cropping Patterns of Annual Crops: A Remote Sensing Review. Remote Sens., 14.","DOI":"10.3390\/rs14102404"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"6553","DOI":"10.1080\/01431161.2019.1569791","article-title":"Crop type classification using a combination of optical and radar remote sensing data: A review","volume":"40","author":"Orynbaikyzy","year":"2019","journal-title":"Int. J. Remote Sens."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Pluto-Kossakowska, J. (2021). Review on Multitemporal Classification Methods of Satellite Images for Crop and Arable Land Recognition. Agriculture, 11.","DOI":"10.3390\/agriculture11100999"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"diab017","DOI":"10.1093\/insilicoplants\/diab017","article-title":"Evolution and application of digital technologies to predict crop type and crop phenology in agriculture","volume":"3","author":"Potgieter","year":"2021","journal-title":"Silico Plants"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Sun, Z., Wang, D., and Zhong, G. (2018, January 6\u20139). A Review of Crop Classification Using Satellite-Based Polarimetric SAR Imagery. Proceedings of the 2018 7th International Conference on Agro-Geoinformatics (Agro-Geoinformatics), Hangzhou, China.","DOI":"10.1109\/Agro-Geoinformatics.2018.8476020"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"473","DOI":"10.1109\/TGRS.1984.6499157","article-title":"A summary of the history of the development of automated remote sensing for agricultural applications","volume":"GE-22","author":"Macdonald","year":"1984","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_7","first-page":"178","article-title":"Integration of optical and SAR remote sensing images for crop-type mapping based on a novel object-oriented feature selection method","volume":"13","author":"Cui","year":"2020","journal-title":"Int. J. Agric. Biol. Eng."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Harfenmeister, K., Itzerott, S., Weltzien, C., and Spengler, D. (2021). Detecting Phenological Development of Winter Wheat and Winter Barley Using Time Series of Sentinel-1 and Sentinel-2. Remote Sens., 13.","DOI":"10.3390\/rs13245036"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"918","DOI":"10.1080\/15481603.2017.1351149","article-title":"Assessing the suitability of data from Sentinel-1A and 2A for crop classification","volume":"54","author":"Sonobe","year":"2017","journal-title":"Gisci.Remote Sens."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Sun, C., Bian, Y., Zhou, T., and Pan, J. (2019). Using of Multi-Source and Multi-Temporal Remote Sensing Data Improves Crop-Type Mapping in the Subtropical Agriculture Region. Sensors, 19.","DOI":"10.3390\/s19102401"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Van Tricht, K., Gobin, A., Gilliams, S., and Piccard, I. (2018). Synergistic Use of Radar Sentinel-1 and Optical Sentinel-2 Imagery for Crop Mapping: A Case Study for Belgium. Remote Sens., 10.","DOI":"10.20944\/preprints201808.0066.v1"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Alberto Garcia-Berna, J., Ouhbi, S., Benmouna, B., Garcia-Mateos, G., Luis Fernandez-Aleman, J., and Miguel Molina-Martinez, J. (2020). Systematic Mapping Study on Remote Sensing in Agriculture. Appl. Sci., 10.","DOI":"10.3390\/app10103456"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"12356","DOI":"10.3390\/rs70912356","article-title":"Assessment of an operational system for crop type map production using high temporal and spatial resolution satellite optical imagery","volume":"7","author":"Inglada","year":"2015","journal-title":"Remote Sens."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Campos-Taberner, M., Garc\u00eda-Haro, F.J., Mart\u00ednez, B., S\u00e1nchez-Ru\u00edz, S., and Gilabert, M.A. (2019). A copernicus sentinel-1 and sentinel-2 classification framework for the 2020+ European common agricultural policy: A case study in Val\u00e8ncia (Spain). Agronomy, 9.","DOI":"10.3390\/agronomy9090556"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"100018","DOI":"10.1016\/j.srs.2021.100018","article-title":"An evaluation of Landsat, Sentinel-2, Sentinel-1 and MODIS data for crop type mapping","volume":"3","author":"Song","year":"2021","journal-title":"Sci. Remote Sens."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Bailly, S., Giordano, S., Landrieu, L., and Chehata, N. (2018, January 22\u201327). Crop-rotation structured classification using multi-source Sentinel images and LPIS for crop type mapping. Proceedings of the IGARSS 2018\u20142018 IEEE International Geoscience and Remote Sensing Symposium, Valencia, Spain.","DOI":"10.1109\/IGARSS.2018.8518427"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1016\/j.rse.2016.10.010","article-title":"Assessing the robustness of Random Forests to map land cover with high resolution satellite image time series over large areas","volume":"187","author":"Pelletier","year":"2016","journal-title":"Remote Sens. Environ."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1016\/j.isprsjprs.2019.01.011","article-title":"DuPLO: A DUal view Point deep Learning architecture for time series classificatiOn","volume":"149","author":"Interdonato","year":"2019","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Ru\u00dfwurm, M., and K\u00f6rner, M. (2018). Multi-temporal land cover classification with sequential recurrent encoders. ISPRS Int. J.-Geo-Inf., 7.","DOI":"10.3390\/ijgi7040129"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Zhao, H., Duan, S., Liu, J., Sun, L., and Reymondin, L. (2021). Evaluation of Five Deep Learning Models for Crop Type Mapping Using Sentinel-2 Time Series Images with Missing Information. Remote Sens., 13.","DOI":"10.3390\/rs13142790"},{"key":"ref_21","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, \u0141., and Polosukhin, I. (2017, January 4\u20139). Attention is all you need. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA."},{"key":"ref_22","unstructured":"Garnot, V.S.F., Landrieu, L., Giordano, S., and Chehata, N. (2020, January 13\u201319). Satellite image time series classification with pixel-set encoders and temporal self-attention. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"421","DOI":"10.1016\/j.isprsjprs.2020.06.006","article-title":"Self-attention for raw optical satellite time series classification","volume":"169","year":"2020","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Ofori-Ampofo, S., Pelletier, C., and Lang, S. (2021). Crop type mapping from optical and radar time series using attention-based deep learning. Remote Sens., 13.","DOI":"10.3390\/rs13224668"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"294","DOI":"10.1016\/j.isprsjprs.2022.03.012","article-title":"Multi-modal temporal attention models for crop mapping from satellite time series","volume":"187","author":"Garnot","year":"2022","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_26","unstructured":"OpenStreetMap Contributors (2022, December 01). Planet Dump. Available online: https:\/\/www.openstreetmap.org."},{"key":"ref_27","unstructured":"German Federal Agency for Cartography and Geodesy (2022, December 01). Verwaltungsgebiete 1:250,000. Data Licence Germany\u2013Attribution\u2013Version 2.0. Available online: https:\/\/gdz.bkg.bund.de\/index.php\/default\/open-data\/verwaltungsgebiete-1-250-000-stand-31-12-vg250-31-12.html."},{"key":"ref_28","unstructured":"Truckenbrodt, J., Cremer, F., Baris, I., and Eberle, J. (2019, January 19\u201320). pyroSAR\u2014A Framework for Large-Scale SAR Satellite Data Processing. Proceedings of the Big Data from Space, Munich, Germany."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Truckenbrodt, J., Freemantle, T., Williams, C., Jones, T., Small, D., Dubois, C., Thiel, C., Rossi, C., Syriou, A., and Giuliani, G. (2019). Towards Sentinel-1 SAR Analysis-Ready Data: A Best Practices Assessment on Preparing Backscatter Data for the Cube. Data, 4.","DOI":"10.3390\/data4030093"},{"key":"ref_30","unstructured":"(2020, October 01). 1TBX\u2014ESA Sentinel-1 Toolbox. v7.0.1. Available online: http:\/\/step.esa.int."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Harfenmeister, K., Spengler, D., and Weltzien, C. (2019). Analyzing Temporal and Spatial Characteristics of Crop Parameters Using Sentinel-1 Backscatter Data. Remote Sens., 11.","DOI":"10.3390\/rs11131569"},{"key":"ref_32","unstructured":"Bohn, N., Scheffler, D., Brell, M., and Segl, K. (2022). SICOR\u2014Sensor Independent Atmospheric Correction, Zenodo."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Hollstein, A., Segl, K., Guanter, L., Brell, M., and Enesco, M. (2016). Ready-to-Use Methods for the Detection of Clouds, Cirrus, Snow, Shadow, Water and Clear Sky Pixels in Sentinel-2 MSI Images. Remote Sens., 8.","DOI":"10.3390\/rs8080666"},{"key":"ref_34","unstructured":"Ministerium f\u00fcr Landwirtschaft, Umwelt und Klimaschutz des Landes Brandenburg (MLUK) (2021, December 20). Daten aus dem Agrarf\u00f6rderantrag. Available online: https:\/\/geobroker.geobasis-bb.de\/gbss.php?MODE=GetProductInformation&PRODUCTID=996f8fd1-c662-4975-b680-3b611fcb5d1f."},{"key":"ref_35","unstructured":"(2022, August 30). Jecam Guidelines for Field Data Collection. Available online: http:\/\/jecam.org\/wp-content\/uploads\/2018\/10\/JECAM_Guidelines_for_Field_Data_Collection_v1_0.pdf."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Inglada, J., Vincent, A., Arias, M., and Marais-Sicre, C. (2016). Improved early crop type identification by joint use of high temporal resolution SAR and optical image time series. Remote Sens., 8.","DOI":"10.3390\/rs8050362"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Asam, S., Gessner, U., Almengor Gonz\u00e1lez, R., Wenzl, M., Kriese, J., and Kuenzer, C. (2022). Mapping Crop Types of Germany by Combining Temporal Statistical Metrics of Sentinel-1 and Sentinel-2 Time Series with LPIS Data. Remote Sens., 14.","DOI":"10.3390\/rs14132981"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Hager, J.W., Fry, L.L., Jacks, S.S., and Hill, D.R. (1992). Datums, Ellipsoids, Grids, and Grid Reference Systems, Defense Mapping Agency Hydrographic\/Topographic Center. Technical Report.","DOI":"10.21236\/ADA247651"},{"key":"ref_39","unstructured":"Bellemans, N., Bontemps, S., Defourny, P., Nicola, L., and Malcorps, P. (2022, September 26). ATBD for L4A Crop Type Mapping. Available online: http:\/\/esa-sen4cap.org\/content\/technical-documents."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"1750","DOI":"10.3389\/fpls.2019.01750","article-title":"A cnn-rnn framework for crop yield prediction","volume":"10","author":"Khaki","year":"2020","journal-title":"Front. Plant Sci."},{"key":"ref_41","unstructured":"Schneider, M., and K\u00f6rner, M. (2022). EuroCrops, Zenodo."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Marszalek, M., L\u00f6sch, M., K\u00f6rner, M., and Schmidhalter, U. (2022). Early crop-type mapping under climate anomalies. Preprints, 2020040316.","DOI":"10.20944\/preprints202004.0316.v2"},{"key":"ref_43","first-page":"1","article-title":"[Re] Satellite Image Time Series Classification with Pixel-Set Encoders and Temporal Self-Attention","volume":"7","author":"Schneider","year":"2021","journal-title":"ReScience C"},{"key":"ref_44","unstructured":"Kondmann, L., Boeck, S., Bonifacio, R., and Zhu, X.X. (2022, January 25\u201329). Early Crop Type Classification with Satellite Imagery\u2014An Empirical Analysis. Proceedings of the ICLR 3rd Workshop on Practical Machine Learning in Developing Countries, Virtual."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1016\/j.isprsjprs.2020.01.001","article-title":"Examining earliest identifiable timing of crops using all available Sentinel 1\/2 imagery and Google Earth Engine","volume":"161","author":"You","year":"2020","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Espinosa-Herrera, J.M., Macedo-Cruz, A., Fern\u00e1ndez-Reynoso, D.S., Flores-Magdaleno, H., Fern\u00e1ndez-Ordo\u00f1ez, Y.M., and Soria-Ru\u00edz, J. (2022). Monitoring and Identification of Agricultural Crops through Multitemporal Analysis of Optical Images and Machine Learning Algorithms. Sensors, 22.","DOI":"10.3390\/s22166106"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Demarez, V., Helen, F., Marais-Sicre, C., and Baup, F. (2019). In-season mapping of irrigated crops using Landsat 8 and Sentinel-1 time series. Remote Sens., 11.","DOI":"10.3390\/rs11020118"},{"key":"ref_48","first-page":"53","article-title":"A progressive crop-type classification using multitemporal remote sensing data and phenological information","volume":"86","author":"Heupel","year":"2018","journal-title":"PFG\u2014J. Photogramm. Remote Sens. Geoinf. Sci."},{"key":"ref_49","unstructured":"Ru\u00dfwurm, M., Tavenard, R., Lef\u00e8vre, S., and K\u00f6rner, M. (2019). Early classification for agricultural monitoring from satellite time series. arXiv."},{"key":"ref_50","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/15\/3\/799\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:19:27Z","timestamp":1760120367000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/15\/3\/799"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,31]]},"references-count":50,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2023,2]]}},"alternative-id":["rs15030799"],"URL":"https:\/\/doi.org\/10.3390\/rs15030799","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,31]]}}}