{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T21:30:01Z","timestamp":1772227801246,"version":"3.50.1"},"reference-count":42,"publisher":"MDPI AG","issue":"24","license":[{"start":{"date-parts":[[2021,12,9]],"date-time":"2021-12-09T00:00:00Z","timestamp":1639008000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1849821"],"award-info":[{"award-number":["1849821"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1951657"],"award-info":[{"award-number":["1951657"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"name":"the United States Department of Agriculture","award":["2021-67021-33446"],"award-info":[{"award-number":["2021-67021-33446"]}]},{"name":"the National Aeronautics and Space Administration","award":["80NSSC21K0946"],"award-info":[{"award-number":["80NSSC21K0946"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Dense time-series remote sensing data with detailed spatial information are highly desired for the monitoring of dynamic earth systems. Due to the sensor tradeoff, most remote sensing systems cannot provide images with both high spatial and temporal resolutions. Spatiotemporal image fusion models provide a feasible solution to generate such a type of satellite imagery, yet existing fusion methods are limited in predicting rapid and\/or transient phenological changes. Additionally, a systematic approach to assessing and understanding how varying levels of temporal phenological changes affect fusion results is lacking in spatiotemporal fusion research. The objective of this study is to develop an innovative hybrid deep learning model that can effectively and robustly fuse the satellite imagery of various spatial and temporal resolutions. The proposed model integrates two types of network models: super-resolution convolutional neural network (SRCNN) and long short-term memory (LSTM). SRCNN can enhance the coarse images by restoring degraded spatial details, while LSTM can learn and extract the temporal changing patterns from the time-series images. To systematically assess the effects of varying levels of phenological changes, we identify image phenological transition dates and design three temporal phenological change scenarios representing rapid, moderate, and minimal phenological changes. The hybrid deep learning model, alongside three benchmark fusion models, is assessed in different scenarios of phenological changes. Results indicate the hybrid deep learning model yields significantly better results when rapid or moderate phenological changes are present. It holds great potential in generating high-quality time-series datasets of both high spatial and temporal resolutions, which can further benefit terrestrial system dynamic studies. The innovative approach to understanding phenological changes\u2019 effect will help us better comprehend the strengths and weaknesses of current and future fusion models.<\/jats:p>","DOI":"10.3390\/rs13245005","type":"journal-article","created":{"date-parts":[[2021,12,9]],"date-time":"2021-12-09T21:46:58Z","timestamp":1639086418000},"page":"5005","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":31,"title":["A Robust Hybrid Deep Learning Model for Spatiotemporal Image Fusion"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4259-4399","authenticated-orcid":false,"given":"Zijun","family":"Yang","sequence":"first","affiliation":[{"name":"Department of Geography and Geographic Information Science, University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA"}]},{"given":"Chunyuan","family":"Diao","sequence":"additional","affiliation":[{"name":"Department of Geography and Geographic Information Science, University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA"}]},{"given":"Bo","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA"}]}],"member":"1968","published-online":{"date-parts":[[2021,12,9]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Zhu, X.L., Cai, F.Y., Tian, J.Q., and Williams, T.K.A. (2018). Spatiotemporal Fusion of Multisource Remote Sensing Data: Literature Survey, Taxonomy, Principles, Applications, and Future Directions. Remote Sens., 10.","DOI":"10.3390\/rs10040527"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1016\/j.rse.2016.11.004","article-title":"Toward mapping crop progress at field scales through fusion of Landsat and MODIS imagery","volume":"188","author":"Gao","year":"2017","journal-title":"Remote Sens. Environ."},{"key":"ref_3","first-page":"63","article-title":"Estimating winter wheat biomass by assimilating leaf area index derived from fusion of Landsat-8 and MODIS data","volume":"49","author":"Dong","year":"2016","journal-title":"Int. J. Appl. Earth Obs. Geoinf."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Gao, F., Anderson, M.C., and Xie, D. (2016, January 10\u201315). Spatial and temporal information fusion for crop condition monitoring. Proceedings of the 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), Beijing, China.","DOI":"10.1109\/IGARSS.2016.7729927"},{"key":"ref_5","first-page":"132","article-title":"Multitemporal fusion of Landsat\/TM and ENVISAT\/MERIS for crop monitoring","volume":"23","author":"Alonso","year":"2013","journal-title":"Int. J. Appl. Earth Obs. Geoinf."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1016\/j.isprsjprs.2019.04.012","article-title":"Innovative pheno-network model in estimating crop phenological stages with satellite time series","volume":"153","author":"Diao","year":"2019","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"111960","DOI":"10.1016\/j.rse.2020.111960","article-title":"Remote sensing phenological monitoring framework to characterize corn and soybean physiological growing stages","volume":"248","author":"Diao","year":"2020","journal-title":"Remote Sens. Environ."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"B\u00e9gu\u00e9, A., Arvor, D., Bellon, B., Betbeder, J., De Abelleyra, D., Ferraz, R.P.D., Lebourgeois, V., Lelong, C., Sim\u00f5es, M., and Ver\u00f3n, S.R. (2018). Remote sensing and cropping practices: A review. Remote Sens., 10.","DOI":"10.3390\/rs10010099"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1613","DOI":"10.1016\/j.rse.2009.03.007","article-title":"A new data fusion model for high spatial-and temporal-resolution mapping of forest disturbance based on Landsat and MODIS","volume":"113","author":"Hilker","year":"2009","journal-title":"Remote Sens. Environ."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1798","DOI":"10.3390\/rs70201798","article-title":"Comparison of Spatiotemporal Fusion Models: A Review","volume":"7","author":"Chen","year":"2015","journal-title":"Remote Sens."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1109\/MGRS.2015.2434351","article-title":"Fusing Landsat and MODIS Data for Vegetation Monitoring","volume":"3","author":"Gao","year":"2015","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"2207","DOI":"10.1109\/TGRS.2006.872081","article-title":"On the blending of the Landsat and MODIS surface reflectance: Predicting daily Landsat surface reflectance","volume":"44","author":"Gao","year":"2006","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1212","DOI":"10.1109\/36.763276","article-title":"Unmixing-based multisensor multiresolution image fusion","volume":"37","author":"Zhukov","year":"1999","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"374","DOI":"10.1016\/j.rse.2016.07.028","article-title":"Land cover change detection by integrating object-based data blending model of Landsat and MODIS","volume":"184","author":"Lu","year":"2016","journal-title":"Remote Sens. Environ."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"24002","DOI":"10.3390\/s150924002","article-title":"Generating daily synthetic Landsat imagery by combining Landsat and MODIS data","volume":"15","author":"Wu","year":"2015","journal-title":"Sensors"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"561","DOI":"10.1080\/2150704X.2013.769283","article-title":"Unified fusion of remote-sensing imagery: Generating simultaneously high-resolution synthetic spatial\u2013temporal\u2013spectral earth observations","volume":"4","author":"Huang","year":"2013","journal-title":"Remote Sens. Lett."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"3190","DOI":"10.3390\/rs5073190","article-title":"Remote sensing based detection of crop phenology for agricultural zones in China using a new threshold method","volume":"5","author":"You","year":"2013","journal-title":"Remote Sens."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"7135","DOI":"10.1109\/TGRS.2016.2596290","article-title":"An integrated framework for the spatio\u2013temporal\u2013spectral fusion of remote sensing images","volume":"54","author":"Shen","year":"2016","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Xue, J., Leung, Y., and Fung, T. (2017). A bayesian data fusion approach to spatio-temporal fusion of remotely sensed images. Remote Sens., 9.","DOI":"10.3390\/rs9121310"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Ke, Y., Im, J., Park, S., and Gong, H. (2016). Downscaling of MODIS One kilometer evapotranspiration using Landsat-8 data and machine learning approaches. Remote Sens., 8.","DOI":"10.3390\/rs8030215"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"3707","DOI":"10.1109\/TGRS.2012.2186638","article-title":"Spatiotemporal reflectance fusion via sparse representation","volume":"50","author":"Huang","year":"2012","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"821","DOI":"10.1109\/JSTARS.2018.2797894","article-title":"Spatiotemporal Satellite Image Fusion Using Deep Convolutional Neural Networks","volume":"11","author":"Song","year":"2018","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1016\/j.rse.2015.11.016","article-title":"A flexible spatiotemporal method for fusing satellite images with different resolutions","volume":"172","author":"Zhu","year":"2016","journal-title":"Remote Sens. Environ."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"2610","DOI":"10.1016\/j.rse.2010.05.032","article-title":"An enhanced spatial and temporal adaptive reflectance fusion model for complex heterogeneous regions","volume":"114","author":"Zhu","year":"2010","journal-title":"Remote Sens. Environ."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1016\/j.rse.2017.10.046","article-title":"Spatio-temporal fusion for daily Sentinel-2 images","volume":"204","author":"Wang","year":"2018","journal-title":"Remote Sens. Environ."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1109\/MGRS.2017.2762307","article-title":"Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources","volume":"5","author":"Zhu","year":"2017","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"166","DOI":"10.1016\/j.isprsjprs.2019.04.015","article-title":"Deep learning in remote sensing applications: A meta-analysis and review","volume":"152","author":"Ma","year":"2019","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"111716","DOI":"10.1016\/j.rse.2020.111716","article-title":"Deep learning in environmental remote sensing: Achievements and challenges","volume":"241","author":"Yuan","year":"2020","journal-title":"Remote Sens. Environ."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Wu, H., and Prasad, S. (2017). Convolutional Recurrent Neural Networks for Hyperspectral Data Classification. Remote Sens., 9.","DOI":"10.3390\/rs9030298"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1016\/j.rse.2018.04.050","article-title":"Urban land-use mapping using a deep convolutional neural network with high spatial resolution multispectral remote sensing imagery","volume":"214","author":"Huang","year":"2018","journal-title":"Remote Sens. Environ."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Teimouri, N., Dyrmann, M., and J\u00f8rgensen, R.N. (2019). A Novel Spatio-Temporal FCN-LSTM Network for Recognizing Various Crop Types Using Multi-Temporal Radar Images. Remote Sens., 11.","DOI":"10.3390\/rs11080990"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Kong, Y.-L., Huang, Q., Wang, C., Chen, J., Chen, J., and He, D. (2018). Long Short-Term Memory Neural Networks for Online Disturbance Detection in Satellite Image Time Series. Remote Sens., 10.","DOI":"10.3390\/rs10030452"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"6552","DOI":"10.1109\/TGRS.2019.2907310","article-title":"StfNet: A Two-Stream Convolutional Neural Network for Spatiotemporal Image Fusion","volume":"57","author":"Liu","year":"2019","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"4273","DOI":"10.1109\/TGRS.2020.3010530","article-title":"Remote Sensing Image Spatiotemporal Fusion Using a Generative Adversarial Network","volume":"59","author":"Zhang","year":"2021","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_37","unstructured":"USDA-NASS (2017). Census of Agriculture."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"341","DOI":"10.1080\/10106049.2011.562309","article-title":"Monitoring US agriculture: The US department of agriculture, national agricultural statistics service, cropland data layer program","volume":"26","author":"Boryan","year":"2011","journal-title":"Geocarto Int."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1109\/TPAMI.2015.2439281","article-title":"Image Super-Resolution Using Deep Convolutional Networks","volume":"38","author":"Dong","year":"2016","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"471","DOI":"10.1016\/S0034-4257(02)00135-9","article-title":"Monitoring vegetation phenology using MODIS","volume":"84","author":"Zhang","year":"2003","journal-title":"Remote Sens. Environ."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Chaithra, C., Taranath, N., Darshan, L., and Subbaraya, C. (2018, January 29\u201331). A Survey on Image Fusion Techniques and Performance Metrics. Proceedings of the 2018 Second International Conference on Electronics, Communication and Aerospace Technology (ICECA), Coimbatore, India.","DOI":"10.1109\/ICECA.2018.8474818"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Arik, S.O., Kliegl, M., Child, R., Hestness, J., Gibiansky, A., Fougner, C., Prenger, R., and Coates, A. (2017). Convolutional recurrent neural networks for small-footprint keyword spotting. arXiv.","DOI":"10.21437\/Interspeech.2017-1737"}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/13\/24\/5005\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:44:20Z","timestamp":1760168660000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/13\/24\/5005"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,9]]},"references-count":42,"journal-issue":{"issue":"24","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["rs13245005"],"URL":"https:\/\/doi.org\/10.3390\/rs13245005","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,12,9]]}}}