{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T18:51:18Z","timestamp":1772823078458,"version":"3.50.1"},"reference-count":39,"publisher":"MDPI AG","issue":"22","license":[{"start":{"date-parts":[[2022,11,18]],"date-time":"2022-11-18T00:00:00Z","timestamp":1668729600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Natural Science Foundation of China","award":["U20B2061"],"award-info":[{"award-number":["U20B2061"]}]},{"name":"National Natural Science Foundation of China","award":["61971233"],"award-info":[{"award-number":["61971233"]}]},{"name":"National Natural Science Foundation of China","award":["SJCX22_0345"],"award-info":[{"award-number":["SJCX22_0345"]}]},{"name":"Research Innovation Program for College Graduates of Jiangsu Province","award":["U20B2061"],"award-info":[{"award-number":["U20B2061"]}]},{"name":"Research Innovation Program for College Graduates of Jiangsu Province","award":["61971233"],"award-info":[{"award-number":["61971233"]}]},{"name":"Research Innovation Program for College Graduates of Jiangsu Province","award":["SJCX22_0345"],"award-info":[{"award-number":["SJCX22_0345"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Short-term precipitation prediction through abundant observation data (ground observation station data, radar data, etc.) is an essential part of the contemporary meteorological prediction system. However, most current studies only use single-modal data, which leads to some problems, such as poor prediction accuracy and little prediction timeliness. This paper proposes a multimodal data fusion precipitation prediction model integrating station data and radar data. Specifically, our model consists of three parts. Firstly, the radar feature encoder comprises a shallow convolution neural network and a stacked convolutional long short term memory network (ConvLSTM), which is used to extract the spatio-temporal features of radar-echo data. The weather station data feature encoder is composed of a fully connected network and an LSTM, which is used to extract the sequential features of the weather station data. Then, the cross-modal feature encoder obtains cross-modal features by aligning and exchanging the feature information of the radar data and the weather station data through the cross-attention mechanism. Finally, the decoder outputs the quantitative short-term precipitation prediction value. Our model can integrate station and radar data characteristics and improve prediction accuracy and timeliness, and can flexibly add other modal features. We have verified our model on four short-term and impending rainfall datasets in South Eastern China, achieving the best performance among the algorithms.<\/jats:p>","DOI":"10.3390\/rs14225839","type":"journal-article","created":{"date-parts":[[2022,11,18]],"date-time":"2022-11-18T04:08:40Z","timestamp":1668744520000},"page":"5839","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["Quantitative Short-Term Precipitation Model Using Multimodal Data Fusion Based on a Cross-Attention Mechanism"],"prefix":"10.3390","volume":"14","author":[{"given":"Yingjie","family":"Cui","sequence":"first","affiliation":[{"name":"School of Artificial Intelligence, Nanjing University of Information Science and Technology, Nanjing 210044, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yunan","family":"Qiu","sequence":"additional","affiliation":[{"name":"School of Electronics and Information Engineering, Nanjing University of Information Science and Technology, Nanjing 210044, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6465-8678","authenticated-orcid":false,"given":"Le","family":"Sun","sequence":"additional","affiliation":[{"name":"School of Computer Science, Nanjing University of Information Science and Technology, Nanjing 210044, China"},{"name":"Jiangsu Collaborative Innovation Center of Atmospheric Environment and Equipment Technology (CICAEET), Nanjing University of Information Science and Technology, Nanjing 210044, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xinyao","family":"Shu","sequence":"additional","affiliation":[{"name":"School of Artificial Intelligence, Nanjing University of Information Science and Technology, Nanjing 210044, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhenyu","family":"Lu","sequence":"additional","affiliation":[{"name":"School of Artificial Intelligence, Nanjing University of Information Science and Technology, Nanjing 210044, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,11,18]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1681","DOI":"10.1109\/JSTARS.2017.2655105","article-title":"Satellite-based nowcasting of extreme rainfall events over Western Himalayan region","volume":"10","author":"Shukla","year":"2017","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens."},{"key":"ref_2","unstructured":"Shi, X., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.k., and Woo, W.c. (2015, January 7\u201312). Convolutional LSTM Network: A machine learning approach for precipitation nowcasting. Proceedings of the 28th International Conference on Neural Information Processing Systems-Volume 1, Montreal, QC, Canada."},{"key":"ref_3","first-page":"26950","article-title":"MAU: A Motion-Aware Unit for Video Prediction and Beyond","volume":"34","author":"Chang","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"166","DOI":"10.1016\/j.neucom.2021.11.021","article-title":"Taylor saves for later: Disentanglement for video prediction using Taylor representation","volume":"472","author":"Pan","year":"2022","journal-title":"Neurocomputing"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"8257","DOI":"10.1109\/TGRS.2020.3042507","article-title":"Recurrent Thrifty Attention Network for Remote Sensing Scene Recognition","volume":"59","author":"Fu","year":"2021","journal-title":"IEEE Trans. Geosci. Remote. Sens."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"2337","DOI":"10.1109\/TGRS.2012.2210429","article-title":"Modeling and prediction of rainfall using radar reflectivity data: A data-mining approach","volume":"51","author":"Kusiak","year":"2012","journal-title":"IEEE Trans. Geosci. Remote. Sens."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"35","DOI":"10.2307\/1268381","article-title":"A three-stage iterative procedure for space-time modeling phillip","volume":"22","author":"Pfeifer","year":"1980","journal-title":"Technometrics"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1000","DOI":"10.1109\/JSTARS.2022.3140442","article-title":"Medium- and Long-Term Precipitation Forecasting Method Based on Data Augmentation and Machine Learning Algorithms","volume":"15","author":"Tang","year":"2022","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"2557","DOI":"10.1109\/TGRS.2018.2874950","article-title":"Estimating Summertime Precipitation from Himawari-8 and Global Forecast System Based on Machine Learning","volume":"57","author":"Min","year":"2019","journal-title":"IEEE Trans. Geosci. Remote. Sens."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"982","DOI":"10.1109\/TGRS.2019.2942280","article-title":"A Machine Learning System for Precipitation Estimation Using Satellite and Ground Radar Network Observations","volume":"58","author":"Chen","year":"2020","journal-title":"IEEE Trans. Geosci. Remote. Sens."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"6347","DOI":"10.1109\/TCSVT.2022.3161103","article-title":"RDEN: Residual Distillation Enhanced Network-Guided Lightweight Synthesized View Quality Enhancement for 3D-HEVC","volume":"32","author":"Pan","year":"2022","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"843","DOI":"10.1109\/JSTARS.2020.3040648","article-title":"PFST-LSTM: A SpatioTemporal LSTM Model With Pseudoflow Prediction for Precipitation Nowcasting","volume":"14","author":"Luo","year":"2021","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Manokij, F., Sarinnapakorn, K., and Vateekul, P. (2019, January 10\u201311). Forecasting Thailand\u2019s Precipitation with Cascading Model of CNN and GRU. Proceedings of the 2019 11th International Conference on Information Technology and Electrical Engineering (ICITEE), Pattaya, Thailand.","DOI":"10.1109\/ICITEED.2019.8929975"},{"key":"ref_14","unstructured":"Kipf, T.N., and Welling, M. (2016). Semi-supervised classification with graph convolutional networks. arXiv."},{"key":"ref_15","first-page":"20","article-title":"Graph attention networks","volume":"1050","author":"Velickovic","year":"2017","journal-title":"Stat"},{"key":"ref_16","first-page":"1024","article-title":"Inductive representation learning on large graphs","volume":"30","author":"Hamilton","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TGRS.2022.3228927","article-title":"Inductive Spatiotemporal Graph Convolutional Networks for Short-Term Quantitative Precipitation Forecasting","volume":"60","author":"Wu","year":"2022","journal-title":"IEEE Trans. Geosci. Remote. Sens."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1007\/s11263-016-0981-7","article-title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations","volume":"123","author":"Krishna","year":"2017","journal-title":"Int. J. Comput. Vis."},{"key":"ref_19","unstructured":"Yu, Z., Cui, Y., Yu, J., Tao, D., and Tian, Q. (2019). Multimodal unified attention networks for vision-and-language interactions. arXiv."},{"key":"ref_20","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Tan, H., and Bansal, M. (2019). Lxmert: Learning cross-modality encoder representations from transformers. arXiv.","DOI":"10.18653\/v1\/D19-1514"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Zhang, W., Han, L., Sun, J., Guo, H., and Dai, J. (2019, January 9\u201312). Application of Multi-channel 3D-cube Successive Convolution Network for Convective Storm Nowcasting. Proceedings of the 2019 IEEE International Conference on Big Data (Big Data), Los Angeles, CA, USA.","DOI":"10.1109\/BigData47090.2019.9005568"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Lin, C.W., and Yang, S. (2021, January 19\u201322). Geospatial-Temporal Convolutional Neural Network for Video-Based Precipitation Intensity Recognition. Proceedings of the 2021 IEEE International Conference on Image Processing (ICIP), Anchorage, AK, USA.","DOI":"10.1109\/ICIP42928.2021.9506728"},{"key":"ref_24","first-page":"5617","article-title":"Deep Learning for Precipitation Nowcasting: A Benchmark and a New Model","volume":"30","author":"Shi","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_25","unstructured":"Wang, Y., Long, M., Wang, J., Gao, Z., and Yu, P.S. (2017, January 4\u20139). PredRNN: Recurrent neural networks for predictive learning using spatiotemporal LSTMs. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_26","unstructured":"Wang, Y., Gao, Z., Long, M., Wang, J., and Philip, S.Y. (2018, January 10\u201315). Predrnn++: Towards a resolution of the deep-in-time dilemma in spatiotemporal predictive learning. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Wang, Y., Zhang, J., Zhu, H., Long, M., Wang, J., and Yu, P.S. (2019, January 15\u201320). Memory in memory: A predictive neural network for learning higher-order non-stationarity from spatiotemporal dynamics. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00937"},{"key":"ref_28","unstructured":"Lin, Z., Li, M., Zheng, Z., Cheng, Y., and Yuan, C. (2020, January 7\u201312). Self-attention convlstm for spatiotemporal prediction. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"25611","DOI":"10.1109\/ACCESS.2020.2970735","article-title":"Deep Learning Local Descriptor for Image Splicing Detection and Localization","volume":"8","author":"Rao","year":"2020","journal-title":"IEEE Access"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TGRS.2022.3231215","article-title":"Multi-Structure KELM With Attention Fusion Strategy for Hyperspectral Image Classification","volume":"60","author":"Sun","year":"2022","journal-title":"IEEE Trans. Geosci. Remote. Sens."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., and Wei, F. (2020, January 23\u201328). Oscar: Object-semantics aligned pre-training for vision-language tasks. Proceedings of the European Conference on Computer Vision, Glasgow, UK.","DOI":"10.1007\/978-3-030-58577-8_8"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"4045","DOI":"10.1109\/JSTARS.2022.3175191","article-title":"SPANet: Successive Pooling Attention Network for Semantic Segmentation of Remote Sensing Images","volume":"15","author":"Sun","year":"2022","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TGRS.2022.3231215","article-title":"Spectral\u2013Spatial Feature Tokenization Transformer for Hyperspectral Image Classification","volume":"60","author":"Sun","year":"2022","journal-title":"IEEE Trans. Geosci. Remote. Sens."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1011","DOI":"10.1109\/JSTARS.2020.2975252","article-title":"Multimodal Bilinear Fusion Network With Second-Order Attention-Based Channel Selection for Land Cover Classification","volume":"13","author":"Li","year":"2020","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens."},{"key":"ref_35","first-page":"5998","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_36","unstructured":"Li, Y., Yu, R., Shahabi, C., and Liu, Y. (2017). Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. arXiv."},{"key":"ref_37","unstructured":"Xu, B., Shen, H., Cao, Q., Qiu, Y., and Cheng, X. (2019). Graph wavelet neural network. arXiv."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Bai, J., Zhu, J., Song, Y., Zhao, L., Hou, Z., Du, R., and Li, H. (2021). A3t-gcn: Attention temporal graph convolutional network for traffic forecasting. ISPRS Int. J. Geo-Inf., 10.","DOI":"10.3390\/ijgi10070485"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Wang, Y., Wu, H., Zhang, J., Gao, Z., Wang, J., Yu, P., and Long, M. (2022). PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning. IEEE Trans. Pattern Anal. Mach. Intell.","DOI":"10.1109\/TPAMI.2022.3165153"}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/14\/22\/5839\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:20:56Z","timestamp":1760145656000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/14\/22\/5839"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,18]]},"references-count":39,"journal-issue":{"issue":"22","published-online":{"date-parts":[[2022,11]]}},"alternative-id":["rs14225839"],"URL":"https:\/\/doi.org\/10.3390\/rs14225839","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,18]]}}}