{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,11]],"date-time":"2026-03-11T17:09:56Z","timestamp":1773248996780,"version":"3.50.1"},"reference-count":62,"publisher":"Association for Computing Machinery (ACM)","issue":"3","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2024,11]]},"abstract":"<jats:p>Advancements in remote sensing technology allowed for collecting vast amounts of satellite and aerial imagery with up to 1 cm pixel resolutions, stored in raster format crucial for various research fields. However, processing this data poses challenges, including resolving data dependencies when location, resolution, and coordinate systems do not align and managing large datasets within memory constraints. This paper introduces RDPro, a novel Spark-based system that efficiently processes and analyzes large raster datasets. RDPro features a new data model tailored for data dependencies in a distributed, shared-nothing environment, complete with tools for loading and writing raster data. It also optimizes core raster operations within Spark, allowing users to integrate complex data science workflows. Comparative analysis shows RDPro outperforms existing systems by up to two orders of magnitude.<\/jats:p>","DOI":"10.14778\/3712221.3712229","type":"journal-article","created":{"date-parts":[[2025,4,7]],"date-time":"2025-04-07T18:03:04Z","timestamp":1744048984000},"page":"613-622","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["RDPro : Distributed Processing of Big Raster Data: [Scalable Data Science]"],"prefix":"10.14778","volume":"18","author":[{"given":"Zhuocheng","family":"Shang","sequence":"first","affiliation":[{"name":"Computer Science and Engineering, University of California, Riverside"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Samriddhi","family":"Singla","sequence":"additional","affiliation":[{"name":"Computer Science and Engineering, University of California, Riverside"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ahmed","family":"Eldawy","sequence":"additional","affiliation":[{"name":"Computer Science and Engineering, University of California, Riverside"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Elia","family":"Scudiero","sequence":"additional","affiliation":[{"name":"USDA-ARS, U.S. Salinity Laboratory, Department of Environmental Sciences, University of California, Riverside, Riverside, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,4,7]]},"reference":[{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.14778\/2536222.2536227"},{"key":"e_1_2_1_3_1","volume-title":"The Rasdaman Array DBMS: Concepts, Architecture, and What People Do With It","author":"Baumann Peter","unstructured":"Peter Baumann. 2022. The Rasdaman Array DBMS: Concepts, Architecture, and What People Do With It. In SSDBM. Association for Computing Machinery, Article 27."},{"key":"e_1_2_1_4_1","volume-title":"SIGMOD.","author":"Baumann Peter","unstructured":"Peter Baumann, Andreas Dehmel, Paula Furtado, Roland Ritsch, and Norbert Widmann. 1998. The Multidimensional Database System RasDaMan. In SIGMOD. Seattle, WA, 575--577."},{"key":"e_1_2_1_5_1","volume-title":"Martin Centre: Cambridge, UK","author":"Daniel Brown","year":"2015","unstructured":"Daniel Brown et al. 2015. Monitoring and evaluating post-disaster recovery using high-resolution satellite imagery-towards standardised indicators for post-disaster recovery. Martin Centre: Cambridge, UK (2015)."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807167.1807271"},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis. 1--11","author":"Buck Joe B","year":"2011","unstructured":"Joe B Buck, Noah Watkins, Jeff LeFevre, Kleoni Ioannidou, Carlos Maltzahn, Neoklis Polyzotis, and Scott Brandt. 2011. Scihadoop: Array-based query processing in hadoop. In Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis. 1--11."},{"key":"e_1_2_1_8_1","volume-title":"Companion of the 2023 International Conference on Management of Data, SIGMOD\/PODS 2023","author":"Chowdhury Kanchan","year":"2023","unstructured":"Kanchan Chowdhury and Mohamed Sarwat. 2023. A Demonstration of GeoTorchAI: A Spatiotemporal Deep Learning Framework. In Companion of the 2023 International Conference on Management of Data, SIGMOD\/PODS 2023, Seattle, WA, USA, June 18-23, 2023. ACM, 195--198."},{"key":"e_1_2_1_9_1","unstructured":"ClimateSpark 2018. ClimateSpark. https:\/\/github.com\/feihugis\/ClimateSpark"},{"key":"e_1_2_1_10_1","doi-asserted-by":"crossref","first-page":"2110","DOI":"10.1002\/hyp.13458","article-title":"Evaluation of uncalibrated energy balance model (BAITSSS) for estimating evapotranspiration in a semiarid, advective climate","volume":"33","author":"Dhungel Ramesh","year":"2019","unstructured":"Ramesh Dhungel, Robert Aiken, Paul D Colaizzi, Xiaomao Lin, Dan O'Brien, R Louis Baumhardt, David K Brauer, and Gary W Marek. 2019. Evaluation of uncalibrated energy balance model (BAITSSS) for estimating evapotranspiration in a semiarid, advective climate. Hydrological Processes 33, 15 (2019), 2110--2130.","journal-title":"Hydrological Processes"},{"key":"e_1_2_1_11_1","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1007\/s00271-023-00874-7","article-title":"Early season irrigation detection and evapotranspiration modeling of winter vegetables based on Planet satellite using water and energy balance algorithm in lower Colorado basin","volume":"42","author":"Dhungel Ramesh","year":"2024","unstructured":"Ramesh Dhungel, Ray G Anderson, Andrew N French, Todd H Skaggs, Mazin Saber, Charles A Sanchez, and Elia Scudiero. 2024. Early season irrigation detection and evapotranspiration modeling of winter vegetables based on Planet satellite using water and energy balance algorithm in lower Colorado basin. Irrigation Science 42, 1 (2024), 15--27.","journal-title":"Irrigation Science"},{"key":"e_1_2_1_12_1","doi-asserted-by":"crossref","unstructured":"Chris Dickens et al. 2019. Defining and quantifying national-level targets indicators and benchmarks for management of natural resources to achieve the sustainable development goals. Sustainability (2019).","DOI":"10.20944\/preprints201811.0611.v1"},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the 2020 ACM SIGMOD international conference on management of data. 1875--1885","author":"Doraiswamy Harish","year":"2020","unstructured":"Harish Doraiswamy and Juliana Freire. 2020. A gpu-friendly geometric data model and algebra for spatial queries. In Proceedings of the 2020 ACM SIGMOD international conference on management of data. 1875--1885."},{"key":"e_1_2_1_14_1","volume-title":"2022 IEEE 38th International Conference on Data Engineering (ICDE). IEEE, 2669--2681","author":"Doraiswamy Harish","year":"2022","unstructured":"Harish Doraiswamy and Juliana Freire. 2022. Spade: Gpu-powered spatial database engine for commodity hardware. In 2022 IEEE 38th International Conference on Data Engineering (ICDE). IEEE, 2669--2681."},{"key":"e_1_2_1_15_1","volume-title":"Spatialhadoop: A MapReduce Framework for Spatial Data. In 2015 IEEE 31st international conference on Data Engineering. IEEE, 1352--1363","author":"Eldawy Ahmed","year":"2015","unstructured":"Ahmed Eldawy and Mohamed F Mokbel. 2015. Spatialhadoop: A MapReduce Framework for Spatial Data. In 2015 IEEE 31st international conference on Data Engineering. IEEE, 1352--1363."},{"key":"e_1_2_1_16_1","volume-title":"Shahed: A MapReduce-based System for Querying and Visualizing Spatio-temporal Satellite Data. In 2015 IEEE 31st international conference on data engineering. IEEE, 1585--1596","author":"Eldawy Ahmed","year":"2015","unstructured":"Ahmed Eldawy, Mohamed F Mokbel, Saif Alharthi, Abdulhadi Alzaidy, Kareem Tarek, and Sohaib Ghani. 2015. Shahed: A MapReduce-based System for Querying and Visualizing Spatio-temporal Satellite Data. In 2015 IEEE 31st international conference on data engineering. IEEE, 1585--1596."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","unstructured":"GDAL\/OGR contributors. 2022. GDAL\/OGR Geospatial Data Abstraction software Library. Open Source Geospatial Foundation. 10.5281\/zenodo.5884351","DOI":"10.5281\/zenodo.5884351"},{"key":"e_1_2_1_18_1","unstructured":"GeoTrellis on Spark 2019. GeoTrellis on Spark. https:\/\/github.com\/wri\/geotrellis-zonal-stats\/blob\/master\/src\/main\/scala\/tutorial\/ZonalStats.scala."},{"key":"e_1_2_1_19_1","first-page":"1258","article-title":"remote sensing for aquaculture","volume":"7","author":"Gernez Pierre","year":"2021","unstructured":"Pierre Gernez, Stephanie CJ Palmer, Yoann Thomas, and Rodney Forster. 2021. remote sensing for aquaculture. Frontiers in Marine Science 7 (2021), 1258.","journal-title":"Frontiers in Marine Science"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/BIGDATA52589.2021.9671569"},{"key":"e_1_2_1_21_1","doi-asserted-by":"crossref","unstructured":"Thomas W Gillespie et al. 2007. Assessment and prediction of natural hazards from satellite imagery. Progress in Physical Geography (2007).","DOI":"10.1177\/0309133307083296"},{"key":"e_1_2_1_22_1","unstructured":"Sean Gillies et al. 2013--. Rasterio: geospatial raster I\/O for Python programmers. Mapbox. https:\/\/github.com\/rasterio\/rasterio"},{"key":"e_1_2_1_23_1","doi-asserted-by":"crossref","unstructured":"Noel Gorelick et al. 2017. Google Earth Engine: Planetary-scale geospatial analysis for everyone. Remote sensing of Environment 202 (2017) 18--27.","DOI":"10.1016\/j.rse.2017.06.031"},{"key":"e_1_2_1_24_1","unstructured":"Rutuja Gurav Het Patel Zhuocheng Shang Ahmed Eldawy Jia Chen Elia Scudiero and Evangelos Papalexakis. 2023. Can SAM recognize crops? Quantifying the zero-shot performance of a semantic segmentation foundation model on generating crop-type maps using satellite imagery for precision agriculture. arXiv:2311.15138 [cs.CV] https:\/\/arxiv.org\/abs\/2311.15138"},{"key":"e_1_2_1_25_1","volume-title":"Hijmans and Jacob van Etten","author":"Robert","year":"2012","unstructured":"Robert J. Hijmans and Jacob van Etten. 2012. raster: Geographic analysis and modeling with raster data. http:\/\/CRAN.R-project.org\/package=raster R package version 2.0-12."},{"key":"e_1_2_1_26_1","volume-title":"ClimateSpark: An in-memory Distributed Computing Framework for Big Climate Data Analytics. Computers & geosciences 115","author":"Hu Fei","year":"2018","unstructured":"Fei Hu, Chaowei Yang, John L Schnase, Daniel Q Duffy, Mengchao Xu, Michael K Bowen, Tsengdar Lee, and Weiwei Song. 2018. ClimateSpark: An in-memory Distributed Computing Framework for Big Climate Data Analytics. Computers & geosciences 115 (2018), 154--166."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ecolind.2013.07.003"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2676536.2676541"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/GEOINFORMATICS.2012.6270312"},{"key":"e_1_2_1_30_1","volume-title":"Chris Mattmann, Brian Wilson, Kim Whitehall, Rishi Verma, Lewis McGibbney, and Paul Ramirez.","author":"Palamuttam Rahul","year":"2015","unstructured":"Rahul Palamuttam, Renato Marroqu\u00edn Mogrovejo, Chris Mattmann, Brian Wilson, Kim Whitehall, Rishi Verma, Lewis McGibbney, and Paul Ramirez. 2015. SciSpark: Applying in-memory Distributed Computing to Weather Event Detection and Tracking. In IEEE BigData. IEEE, 2020--2026."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.14778\/3025111.3025117"},{"key":"e_1_2_1_32_1","volume-title":"Satellite remote sensing and the management of natural resources","author":"Pettorelli Nathalie","unstructured":"Nathalie Pettorelli. 2019. Satellite remote sensing and the management of natural resources. Oxford University Press."},{"key":"e_1_2_1_33_1","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1007\/s12524-013-0312-1","article-title":"Modelling changes in land cover patterns in Mtunzini, South Africa using satellite imagery","volume":"42","author":"Kamleshan Pillay","year":"2014","unstructured":"Kamleshan Pillay et al. 2014. Modelling changes in land cover patterns in Mtunzini, South Africa using satellite imagery. Journal of the Indian Society of Remote Sensing 42, 1 (2014), 51--60.","journal-title":"Journal of the Indian Society of Remote Sensing"},{"key":"e_1_2_1_34_1","unstructured":"PostGIS 2022. PostGIS. https:\/\/postgis.net\/."},{"key":"e_1_2_1_35_1","volume-title":"QGIS Geographic Information System","author":"Development Team QGIS","unstructured":"QGIS Development Team. 2022. QGIS Geographic Information System. QGIS Association. https:\/\/www.qgis.org"},{"key":"e_1_2_1_36_1","volume-title":"Environmental Systems Research Institute Redlands","author":"CA","year":"2022","unstructured":"CA: Environmental Systems Research Institute Redlands. 2022. ArcGIS. https:\/\/www.esri.com\/en-us\/arcgis\/about-arcgis\/overview"},{"key":"e_1_2_1_37_1","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1016\/j.geodrs.2014.10.004","article-title":"Regional scale soil salinity evaluation using Landsat 7, western San Joaquin Valley, California","volume":"2","author":"Scudiero Elia","year":"2014","unstructured":"Elia Scudiero, Todd H Skaggs, and Dennis L Corwin. 2014. Regional scale soil salinity evaluation using Landsat 7, western San Joaquin Valley, California, USA. Geoderma Regional 2 (2014), 82--90.","journal-title":"USA. Geoderma Regional"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.15482\/USDA.ADC\/1227096"},{"key":"e_1_2_1_39_1","volume-title":"Exploring Google Earth Engine platform for big data processing: Classification of multi-temporal satellite imagery for crop mapping. frontiers in Earth Science 5","author":"Shelestov Andrii","year":"2017","unstructured":"Andrii Shelestov, Mykola Lavreniuk, Nataliia Kussul, Alexei Novikov, and Sergii Skakun. 2017. Exploring Google Earth Engine platform for big data processing: Classification of multi-temporal satellite imagery for crop mapping. frontiers in Earth Science 5 (2017), 17."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.14778\/3352063.3352107"},{"key":"e_1_2_1_41_1","volume-title":"Conference on Neural Information Processing Systems Track on Datasets and Benchmarks.","author":"Singla Samriddhi","year":"2021","unstructured":"Samriddhi Singla, Ayan Mukhopadhyay, Michael Wilbur, Tina Diao, Vinayak Gajjewar, Ahmed Eldawy, Mykel Kochenderfer, Ross Shachter, and Abhishek Dubey. 2021. Wildfiredb: An open-source dataset connecting wildfire spread with relevant determinants. In Conference on Neural Information Processing Systems Track on Datasets and Benchmarks."},{"key":"e_1_2_1_42_1","doi-asserted-by":"crossref","first-page":"61","DOI":"10.14311\/gi.15.1.5","article-title":"GPU-accelerated raster map reprojection","volume":"15","author":"Sloup Petr","year":"2016","unstructured":"Petr Sloup. 2016. GPU-accelerated raster map reprojection. Geoinformatics FCE CTU 15, 1 (2016), 61--68.","journal-title":"Geoinformatics FCE CTU"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cageo.2011.11.012"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2013.19"},{"key":"e_1_2_1_45_1","volume-title":"SciDB: A Database Management System for Applications with Complex Analytics. Computing in Science Engineering 15 (05","author":"Stonebraker Michael","year":"2013","unstructured":"Michael Stonebraker, Paul Brown, Donghui Zhang, and Jacek Becla. 2013. SciDB: A Database Management System for Applications with Complex Analytics. Computing in Science Engineering 15 (05 2013), 54--62."},{"key":"e_1_2_1_46_1","unstructured":"U.S. Geological Survey. 2024. Landsat 8. https:\/\/www.usgs.gov\/landsat-missions\/landsat-8"},{"key":"e_1_2_1_47_1","unstructured":"Planet Team. 2018--. Planet Application Program Interface: In Space for Life on Earth. https:\/\/api.planet.com"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/MDM52706.2021.00024"},{"key":"e_1_2_1_49_1","unstructured":"TileDB Inc. 2023. tiledb: Universal Storage Engine for Sparse and Dense Multidimensional Arrays. https:\/\/github.com\/TileDB-Inc\/TileDB-R R package version 0.20.1."},{"key":"e_1_2_1_50_1","volume-title":"Proceedings of the VLDB Endowment 11","author":"Zacharatou Eleni Tzirita","year":"2017","unstructured":"Eleni Tzirita Zacharatou, Harish Doraiswamy, Anastasia Ailamaki, Claudio Silva, and Juliana Freire. 2017. GPU rasterization for real-time spatial aggregation over arbitrary polygons. Proceedings of the VLDB Endowment 11, 3 (2017)."},{"key":"e_1_2_1_51_1","unstructured":"USGS 2024. Earth Observing Satellites. https:\/\/www.usgs.gov\/calval\/earth-observing-satellites."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/2666310.2666387"},{"key":"e_1_2_1_53_1","volume-title":"Geoinformatics 2007: Geospatial Information Technology and Applications","author":"Wu Ye","unstructured":"Ye Wu, Ying Ge, Weibao Yan, and Xinyu Li. 2007. Improving the performance of spatial raster analysis in GIS using GPU. In Geoinformatics 2007: Geospatial Information Technology and Applications, Vol. 6754. SPIE, 244--254."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2915237"},{"key":"e_1_2_1_55_1","doi-asserted-by":"crossref","first-page":"582","DOI":"10.1109\/JPROC.2012.2196249","article-title":"Using high-resolution airborne and satellite imagery to assess crop growth and yield variability for precision agriculture","volume":"101","author":"Chenghai Yang","year":"2012","unstructured":"Chenghai Yang et al. 2012. Using high-resolution airborne and satellite imagery to assess crop growth and yield variability for precision agriculture. Proc. IEEE 101, 3 (2012), 582--592.","journal-title":"Proc. IEEE"},{"key":"e_1_2_1_56_1","doi-asserted-by":"crossref","unstructured":"Jia Yu Jinxuan Wu and Mohamed Sarwat. 2015. GeoSpark: A cluster computing framework for processing large-scale spatial data. In SIGSPATIAL.","DOI":"10.1145\/2820783.2820860"},{"key":"e_1_2_1_57_1","volume-title":"2016 IEEE 32nd International Conference on Data Engineering (ICDE). IEEE, 1410--1413","author":"Yu Jia","year":"2016","unstructured":"Jia Yu, Jinxuan Wu, and Mohamed Sarwat. 2016. A Demonstration of GeoSpark: A Cluster Computing Framework for Processing Big Spatial Data. In 2016 IEEE 32nd International Conference on Data Engineering (ICDE). IEEE, 1410--1413."},{"key":"e_1_2_1_58_1","volume-title":"Spatial Data Management in Apache Spark: The GeoSpark Perspective and Beyond. GeoInformatica 23","author":"Yu Jia","year":"2019","unstructured":"Jia Yu, Zongsi Zhang, and Mohamed Sarwat. 2019. Spatial Data Management in Apache Spark: The GeoSpark Perspective and Beyond. GeoInformatica 23 (2019)."},{"key":"e_1_2_1_59_1","doi-asserted-by":"crossref","first-page":"1247","DOI":"10.14778\/3231751.3231754","article-title":"ChronosDB: Distributed, File based","volume":"11","author":"Rodriges Zalipynis Ramon Antonio","year":"2018","unstructured":"Ramon Antonio Rodriges Zalipynis. 2018. ChronosDB: Distributed, File based, Geospatial Array DBMS. Proceedings of the VLDB Endowment 11, 10 (2018), 1247--1261.","journal-title":"Geospatial Array DBMS. Proceedings of the VLDB Endowment"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/2070770.2070772"},{"key":"e_1_2_1_61_1","volume-title":"Proceedings of the 3rd ACM SIGSPATIAL International Workshop on GeoStreaming","author":"Zhang Jianting","year":"2012","unstructured":"Jianting Zhang and Simin You. 2012. CudaGIS: report on the design and realization of a massive data parallel GIS on GPUs. In Proceedings of the 3rd ACM SIGSPATIAL International Workshop on GeoStreaming (Redondo Beach, California) (IWGS '12). Association for Computing Machinery, New York, NY, USA, 101--108."},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766196.2766201"},{"key":"e_1_2_1_63_1","volume-title":"Deep learning for remote sensing data: A technical tutorial on the state of the art","author":"Zhang Liangpei","year":"2016","unstructured":"Liangpei Zhang, Lefei Zhang, and Bo Du. 2016. Deep learning for remote sensing data: A technical tutorial on the state of the art. IEEE Geoscience and remote sensing magazine 4, 2 (2016), 22--40."}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3712221.3712229","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,4,7]],"date-time":"2025-04-07T18:30:42Z","timestamp":1744050642000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3712221.3712229"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11]]},"references-count":62,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2024,11]]}},"alternative-id":["10.14778\/3712221.3712229"],"URL":"https:\/\/doi.org\/10.14778\/3712221.3712229","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2024,11]]},"assertion":[{"value":"2025-04-07","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}