{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,10]],"date-time":"2026-01-10T18:59:49Z","timestamp":1768071589175,"version":"3.49.0"},"reference-count":34,"publisher":"MDPI AG","issue":"9","license":[{"start":{"date-parts":[[2021,4,23]],"date-time":"2021-04-23T00:00:00Z","timestamp":1619136000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Remotely-sensed satellite image fusion is indispensable for the generation of long-term gap-free Earth observation data. While cloud computing (CC) provides the big picture for RS big data (RSBD), the fundamental question of the efficient fusion of RSBD on CC platforms has not yet been settled. To this end, we propose a lightweight cloud-native framework for the elastic processing of RSBD in this study. With the scaling mechanisms provided by both the Infrastructure as a Service (IaaS) and Platform as a Services (PaaS) of CC, the Spark-on-Kubernetes operator model running in the framework can enhance the efficiency of Spark-based algorithms without considering bottlenecks such as task latency caused by an unbalanced workload, and can ease the burden to tune the performance parameters for their parallel algorithms. Internally, we propose a task scheduling mechanism (TSM) to dynamically change the Spark executor pods\u2019 affinities to the computing hosts. The TSM learns the workload of a computing host. Learning from the ratio between the number of completed and failed tasks on a computing host, the TSM dispatches Spark executor pods to newer and less-overwhelmed computing hosts. In order to illustrate the advantage, we implement a parallel enhanced spatial and temporal adaptive reflectance fusion model (PESTARFM) to enable the efficient fusion of big RS images with a Spark aggregation function. We construct an OpenStack cloud computing environment to test the usability of the framework. According to the experiments, TSM can improve the performance of the PESTARFM using only PaaS scaling to about 11.7%. When using both the IaaS and PaaS scaling, the maximum performance gain with the TSM can be even greater than 13.6%. The fusion of such big Sentinel and PlanetScope images requires less than 4 min in the experimental environment.<\/jats:p>","DOI":"10.3390\/s21092971","type":"journal-article","created":{"date-parts":[[2021,4,25]],"date-time":"2021-04-25T02:12:57Z","timestamp":1619316777000},"page":"2971","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["On-the-Fly Fusion of Remotely-Sensed Big Data Using an Elastic Computing Paradigm with a Containerized Spark Engine on Kubernetes"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1696-1503","authenticated-orcid":false,"given":"Wei","family":"Huang","sequence":"first","affiliation":[{"name":"School of Civil and Hydraulic Engineering, Huazhong University of Science and Technology, Wuhan 430074, China"}]},{"given":"Jianzhong","family":"Zhou","sequence":"additional","affiliation":[{"name":"School of Civil and Hydraulic Engineering, Huazhong University of Science and Technology, Wuhan 430074, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2488-7243","authenticated-orcid":false,"given":"Dongying","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Civil and Hydraulic Engineering, Huazhong University of Science and Technology, Wuhan 430074, China"}]}],"member":"1968","published-online":{"date-parts":[[2021,4,23]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"208","DOI":"10.1038\/nrg.2017.113","article-title":"Cloud computing for genomic data analysis and collaboration","volume":"19","author":"Langmead","year":"2018","journal-title":"Nat. Rev. Genet."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"325","DOI":"10.1016\/j.isprsjprs.2018.07.017","article-title":"A 30-m landsat-derived cropland extent product of Australia and China using random forest machine learning algorithm on Google Earth Engine cloud computing platform","volume":"144","author":"Teluguntla","year":"2018","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1154","DOI":"10.1016\/j.future.2017.02.044","article-title":"A cloud-based remote sensing data production system","volume":"86","author":"Yan","year":"2018","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1016\/j.inffus.2016.03.003","article-title":"A review of remote sensing image fusion methods","volume":"32","author":"Ghassemian","year":"2016","journal-title":"Inf. Fusion"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"4294","DOI":"10.1109\/TGRS.2018.2890513","article-title":"An Efficient and Scalable Framework for Processing Remotely Sensed Big Data in Cloud Computing Environments","volume":"57","author":"Sun","year":"2019","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Tian, F., Wu, B., Zeng, H., Zhang, X., and Xu, J. (2019). Efficient Identification of Corn Cultivation Area with Multitemporal Synthetic Aperture Radar and Optical Images in the Google Earth Engine Cloud Platform. Remote Sens., 11.","DOI":"10.3390\/rs11060629"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1341","DOI":"10.1109\/TC.2017.2669964","article-title":"Cross-Platform Resource Scheduling for Spark and MapReduce on YARN","volume":"66","author":"Cheng","year":"2017","journal-title":"IEEE Trans. Comput."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1170","DOI":"10.1016\/j.ins.2019.10.035","article-title":"A scheduling scheme in the cloud computing environment using deep Q-learning","volume":"512","author":"Tong","year":"2020","journal-title":"Inf. Sci."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2797211","article-title":"Allocation of Virtual Machines in Cloud Data Centers\u2014A Survey of Problem Models and Optimization Algorithms","volume":"48","author":"Mann","year":"2015","journal-title":"ACM Comput. Surv."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Bhimani, J., Yang, Z., Leeser, M., and Mi, N. (2017, January 12\u201314). Accelerating big data applications using lightweight virtualization framework on enterprise cloud. Proceedings of the 2017 IEEE High Performance Extreme Computing Conference (HPEC), Institute of Electrical and Electronics Engineers (IEEE), Waltham, MA, USA.","DOI":"10.1109\/HPEC.2017.8091086"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"3566","DOI":"10.1109\/TII.2020.3022843","article-title":"Evaluating Docker for Lightweight Virtualization of Distributed and Time-Sensitive Applications in Industrial Automation","volume":"17","author":"Sollfrank","year":"2021","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Baresi, L., Denaro, G., and Quattrocchi, G. (2019, January 27\u201330). Big-Data Applications as Self-Adaptive Systems of Systems. Proceedings of the 2019 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW), Berlin, Germany.","DOI":"10.1109\/ISSREW.2019.00066"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"197","DOI":"10.1016\/j.future.2019.04.008","article-title":"SWITCH workbench: A novel approach for the development and deployment of time-critical microservice-based cloud-native applications","volume":"99","author":"Cigale","year":"2019","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"112002","DOI":"10.1016\/j.rse.2020.112002","article-title":"A summary of the special issue on remote sensing of land change science with Google earth engine","volume":"248","author":"Wang","year":"2020","journal-title":"Remote Sens. Environ."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1016\/j.inffus.2020.12.008","article-title":"A spatial-channel progressive fusion ResNet for remote sensing classification","volume":"70","author":"Zhu","year":"2021","journal-title":"Inf. Fusion"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1109\/MGRS.2016.2637824","article-title":"Hyperspectral and Multispectral Data Fusion: A comparative review of the recent literature","volume":"5","author":"Yokoya","year":"2017","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1038\/s41592-019-0650-1","article-title":"CLIJ: GPU-accelerated image processing for everyone","volume":"17","author":"Haase","year":"2020","journal-title":"Nat. Methods"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Jia, D., Cheng, C., Song, C., Shen, S., Ning, L., and Zhang, T. (2021). A Hybrid Deep Learning-Based Spatiotemporal Fusion Method for Combining Satellite Images with Different Resolutions. Remote Sens., 13.","DOI":"10.3390\/rs13040645"},{"key":"ref_19","unstructured":"Hong, S., Choi, J., and Jeong, W.K. (2020). Distributed Interactive Visualization using GPU-Optimized Spark. IEEE Trans. Vis. Comput. Graph., 1\u201315."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Baresi, L., and Quattrocchi, G. (2018, January 27\u201328). Towards Vertically Scalable Spark Applications. Proceedings of the Euro-Par 2018: Parallel Processing Workshops, Turin, Italy.","DOI":"10.1007\/978-3-030-10549-5_9"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"e5323","DOI":"10.1002\/cpe.5323","article-title":"A study on the evaluation of HPC microservices in containerized environment","volume":"33","author":"Jha","year":"2021","journal-title":"Concurr. Comput. Pract. Exp."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Xu, C., Du, X., Yan, Z., and Fan, X. (2020). ScienceEarth: A Big Data Platform for Remote Sensing Data Processing. Remote Sens., 12.","DOI":"10.3390\/rs12040607"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1109\/JSTARS.2016.2547020","article-title":"In-Memory Parallel Processing of Massive Remotely Sensed Data Using an Apache Spark on Hadoop YARN Model","volume":"10","author":"Huang","year":"2017","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Buzachis, A., Galletta, A., Carnevale, L., Celesti, A., Fazio, M., and Villari, M. (2018, January 1\u20133). Towards Osmotic Computing: Analyzing Overlay Network Solutions to Optimize the Deployment of Container-Based Microservices in Fog, Edge and IoT Environments. Proceedings of the 2018 IEEE 2nd International Conference on Fog and Edge Computing (ICFEC), Washinton, DC, USA.","DOI":"10.1109\/CFEC.2018.8358729"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Suo, K., Zhao, Y., Chen, W., and Rao, J. (2018, January 16\u201319). An Analysis and Empirical Study of Container Networks. Proceedings of the IEEE INFOCOM 2018-IEEE Conference on Computer Communications, Honolulu, HI, USA.","DOI":"10.1109\/INFOCOM.2018.8485865"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Armbrust, M., Xin, R.S., Lian, C., Huai, Y., Liu, D., Bradley, J.K., Meng, X., Kaftan, T., Franklin, M.J., and Ghodsi, A. (June, January 31). Spark SQL: Relational Data Processing in Spark. Proceedings of the Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, Melbourne, Australia.","DOI":"10.1145\/2723372.2742797"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"2610","DOI":"10.1016\/j.rse.2010.05.032","article-title":"An enhanced spatial and temporal adaptive reflectance fusion model for complex heterogeneous regions","volume":"114","author":"Zhu","year":"2010","journal-title":"Remote Sens. Environ."},{"key":"ref_28","first-page":"38","article-title":"OpenStack: Toward an open-source solution for cloud computing","volume":"55","author":"Sefraoui","year":"2012","journal-title":"Int. J. Comput. Appl."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Meireles, F., and Malheiro, B. (2014, January 25\u201329). Integrated Management of IaaS Resources. Proceedings of the Euro-Par 2014: Parallel Processing Workshops, Berlin, Germany.","DOI":"10.1007\/978-3-319-14313-2_7"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"152","DOI":"10.1016\/j.isprsjprs.2020.04.001","article-title":"Google Earth Engine for geo-big data applications: A meta-analysis and systematic review","volume":"164","author":"Tamiminia","year":"2020","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"353","DOI":"10.1016\/j.future.2016.06.009","article-title":"pipsCloud: High performance cloud computing for remote sensing big data management and processing","volume":"78","author":"Wang","year":"2018","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Kang, S., and Lee, K. (2016). Auto-Scaling of Geo-Based Image Processing in an OpenStack Cloud Computing Environment. Remote Sens., 8.","DOI":"10.3390\/rs8080662"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Huang, W., Zhang, W., Zhang, D., and Meng, L. (2017). Elastic Spatial Query Processing in OpenStack Cloud Computing Environment for Time-Constraint Data Analysis. ISPRS Int. J. Geo Inf., 6.","DOI":"10.3390\/ijgi6030084"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3381027","article-title":"A Survey on Automatic Parameter Tuning for Big Data Processing Systems","volume":"53","author":"Herodotou","year":"2020","journal-title":"ACM Comput. Surv."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/9\/2971\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:52:03Z","timestamp":1760161923000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/9\/2971"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,23]]},"references-count":34,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2021,5]]}},"alternative-id":["s21092971"],"URL":"https:\/\/doi.org\/10.3390\/s21092971","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,4,23]]}}}