{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,18]],"date-time":"2026-07-18T15:57:50Z","timestamp":1784390270874,"version":"3.55.0"},"reference-count":52,"publisher":"Association for Computing Machinery (ACM)","issue":"12","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2020,8]]},"abstract":"<jats:p>Cloud object stores such as Amazon S3 are some of the largest and most cost-effective storage systems on the planet, making them an attractive target to store large data warehouses and data lakes. Unfortunately, their implementation as key-value stores makes it difficult to achieve ACID transactions and high performance: metadata operations such as listing objects are expensive, and consistency guarantees are limited. In this paper, we present Delta Lake, an open source ACID table storage layer over cloud object stores initially developed at Databricks. Delta Lake uses a transaction log that is compacted into Apache Parquet format to provide ACID properties, time travel, and significantly faster metadata operations for large tabular datasets (e.g., the ability to quickly search billions of table partitions for those relevant to a query). It also leverages this design to provide high-level features such as automatic data layout optimization, upserts, caching, and audit logs. Delta Lake tables can be accessed from Apache Spark, Hive, Presto, Redshift and other systems. Delta Lake is deployed at thousands of Databricks customers that process exabytes of data per day, with the largest instances managing exabyte-scale datasets and billions of objects.<\/jats:p>","DOI":"10.14778\/3415478.3415560","type":"journal-article","created":{"date-parts":[[2020,9,14]],"date-time":"2020-09-14T18:46:40Z","timestamp":1600109200000},"page":"3411-3424","source":"Crossref","is-referenced-by-count":226,"title":["Delta lake"],"prefix":"10.14778","volume":"13","author":[{"given":"Michael","family":"Armbrust","sequence":"first","affiliation":[{"name":"Databricks"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tathagata","family":"Das","sequence":"additional","affiliation":[{"name":"Databricks"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Liwen","family":"Sun","sequence":"additional","affiliation":[{"name":"Databricks"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Burak","family":"Yavuz","sequence":"additional","affiliation":[{"name":"Databricks"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shixiong","family":"Zhu","sequence":"additional","affiliation":[{"name":"Databricks"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mukul","family":"Murthy","sequence":"additional","affiliation":[{"name":"Databricks"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joseph","family":"Torres","sequence":"additional","affiliation":[{"name":"Databricks"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Herman","family":"van Hovell","sequence":"additional","affiliation":[{"name":"Databricks"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Adrian","family":"Ionescu","sequence":"additional","affiliation":[{"name":"Databricks"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alicja","family":"\u0141uszczak","sequence":"additional","affiliation":[{"name":"Databricks"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Micha\u0142","family":"\u015awitakowski","sequence":"additional","affiliation":[{"name":"Databricks"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Micha\u0142","family":"Szafra\u0144ski","sequence":"additional","affiliation":[{"name":"Databricks"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiao","family":"Li","sequence":"additional","affiliation":[{"name":"Databricks"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Takuya","family":"Ueshin","sequence":"additional","affiliation":[{"name":"Databricks"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mostafa","family":"Mokhtar","sequence":"additional","affiliation":[{"name":"Databricks"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peter","family":"Boncz","sequence":"additional","affiliation":[{"name":"CWI"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ali","family":"Ghodsi","sequence":"additional","affiliation":[{"name":"UC Berkeley"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sameer","family":"Paranjpye","sequence":"additional","affiliation":[{"name":"Databricks"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pieter","family":"Senster","sequence":"additional","affiliation":[{"name":"Databricks"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Reynold","family":"Xin","sequence":"additional","affiliation":[{"name":"Databricks"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matei","family":"Zaharia","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,8]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Amazon Athena. https:\/\/aws.amazon.com\/athena\/."},{"key":"e_1_2_1_2_1","unstructured":"Amazon Kinesis. https:\/\/aws.amazon.com\/kinesis\/."},{"key":"e_1_2_1_3_1","unstructured":"Amazon Redshift. https:\/\/aws.amazon.com\/redshift\/."},{"key":"e_1_2_1_4_1","unstructured":"Amazon S3. https:\/\/aws.amazon.com\/s3\/."},{"key":"e_1_2_1_5_1","unstructured":"Apache Hadoop. https:\/\/hadoop.apache.org."},{"key":"e_1_2_1_6_1","unstructured":"Apache Kudu. https:\/\/kudu.apache.org."},{"key":"e_1_2_1_7_1","unstructured":"Apache HBase. https:\/\/hbase.apache.org."},{"key":"e_1_2_1_8_1","unstructured":"Apache Hudi. https:\/\/hudi.apache.org."},{"key":"e_1_2_1_9_1","unstructured":"Apache Hudi GitHub issue: Future support for multi-client concurrent write? https:\/\/github.com\/apache\/incubator-hudi\/issues\/1240."},{"key":"e_1_2_1_10_1","unstructured":"Apache Iceberg. https:\/\/iceberg.apache.org."},{"key":"e_1_2_1_11_1","unstructured":"Apache Kafka. https:\/\/kafka.apache.org."},{"key":"e_1_2_1_12_1","unstructured":"Apache ORC. https:\/\/orc.apache.org."},{"key":"e_1_2_1_13_1","unstructured":"Apache Parquet. https:\/\/parquet.apache.org."},{"key":"e_1_2_1_14_1","volume-title":"SIGMOD, page 601--613","author":"Armbrust M.","year":"2018","unstructured":"M. Armbrust, T. Das, J. Torres, B. Yavuz, S. Zhu, R. Xin, A. Ghodsi, I. Stoica, and M. Zaharia. Structured streaming: A declarative API for real-time applications in Apache Spark. In SIGMOD, page 601--613, New York, NY, USA, 2018. Association for Computing Machinery."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1721654.1721672"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2742797"},{"key":"e_1_2_1_17_1","unstructured":"Azure Blob Storage. https:\/\/https:\/\/azure.microsoft.com\/en-us\/services\/storage\/blobs\/."},{"key":"e_1_2_1_18_1","unstructured":"Azure Data Lake Storage. https:\/\/azure.microsoft.com\/en-us\/services\/storage\/data-lake-storage\/."},{"key":"e_1_2_1_19_1","first-page":"761","volume-title":"Bolt-on causal consistency","author":"Bailis P.","year":"2013","unstructured":"P. Bailis, A. Ghodsi, J. Hellerstein, and I. Stoica. Bolt-on causal consistency. pages 761--772, 06 2013."},{"key":"e_1_2_1_20_1","first-page":"251","volume-title":"Building a database on S3","author":"Brantner M.","year":"2008","unstructured":"M. Brantner, D. Florescu, D. Graf, D. Kossmann, and T. Kraska. Building a database on S3. pages 251--264, 01 2008."},{"key":"e_1_2_1_21_1","unstructured":"A. Conway and J. Minnick. Introducing Delta Engine. https:\/\/databricks.com\/blog\/2020\/06\/24\/introducing-delta-engine.html."},{"key":"e_1_2_1_22_1","first-page":"235","volume-title":"CIDR","author":"Curino C.","year":"2011","unstructured":"C. Curino, E. Jones, R. Popa, N. Malviya, E. Wu, S. Madden, H. Balakrishnan, and N. Zeldovich. Relational cloud: A database-as-a-service for the cloud. In CIDR, pages 235--240, 04 2011."},{"key":"e_1_2_1_23_1","first-page":"215","volume-title":"The Snowflake elastic data warehouse","author":"Dageville B.","year":"2016","unstructured":"B. Dageville, J. Huang, A. Lee, A. Motivala, A. Munir, S. Pelley, P. Povinec, G. Rahn, S. Triantafyllis, P. Unterbrunner, T. Cruanes, M. Zukowski, V. Antonov, A. Avanes, J. Bock, J. Claybaugh, D. Engovatov, and M. Hentschel. The Snowflake elastic data warehouse. pages 215--226, 06 2016."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btr330"},{"key":"e_1_2_1_25_1","unstructured":"Databricks runtime. https:\/\/databricks.com\/product\/databricks-runtime."},{"key":"e_1_2_1_26_1","unstructured":"Delta Lake website. https:\/\/delta.io."},{"key":"e_1_2_1_27_1","volume-title":"Official Journal of the European Union, 59:1--88","author":"General Data Protection Regulation. Regulation (EU) 2016\/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95\/46.","year":"2016","unstructured":"General Data Protection Regulation. Regulation (EU) 2016\/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95\/46. Official Journal of the European Union, 59:1--88, 2016."},{"key":"e_1_2_1_28_1","unstructured":"Glow: An open-source toolkit for large-scale genomic analysis. https:\/\/projectglow.io."},{"key":"e_1_2_1_29_1","unstructured":"Google BigQuery. https:\/\/cloud.google.com\/bigquery."},{"key":"e_1_2_1_30_1","unstructured":"Google Cloud Storage. https:\/\/cloud.google.com\/storage."},{"key":"e_1_2_1_31_1","unstructured":"Google Cloud Storage consistency documentation. https:\/\/cloud.google.com\/storage\/docs\/consistency."},{"key":"e_1_2_1_32_1","unstructured":"Hive 3 ACID documentation from Cloudera. https:\/\/docs.cloudera.com\/HDPDocuments\/HDP3\/HDP-3.1.5\/using-hiveql\/content\/hive_3_internals.html."},{"key":"e_1_2_1_33_1","unstructured":"H. Jaani. New data ingestion network for Databricks: The partner ecosystem for applications database and big data integrations into Delta Lake. https:\/\/databricks.com\/blog\/2020\/02\/24\/new-databricks-data-ingestion-network-for-applications-database-and-big-data-integrations-into-delta-lake.html 2020."},{"key":"e_1_2_1_34_1","volume-title":"Aug.","author":"Li H.","year":"2009","unstructured":"H. Li, B. Handsaker, A. Wysoker, T. Fennell, J. Ruan, N. Homer, G. Marth, G. Abecasis, R. Durbin, and 1000 Genome Project Data Processing Subgroup. The sequence alignment\/map format and SAMtools. Bioinformatics, 25(16):2078--2079, Aug. 2009."},{"key":"e_1_2_1_35_1","unstructured":"G. M. Morton. A computer oriented geodetic data base; and a new technique in file sequencing. IBM Technical Report 1966."},{"key":"e_1_2_1_36_1","volume-title":"Small files, big foils: Addressing the associated metadata and application challenges. https:\/\/blog.cloudera.com\/small-files-big-foils-addressing-the-associated-metadata-and-application-challenges\/","author":"Naik S.","year":"2019","unstructured":"S. Naik and B. Gummalla. Small files, big foils: Addressing the associated metadata and application challenges. https:\/\/blog.cloudera.com\/small-files-big-foils-addressing-the-associated-metadata-and-application-challenges\/, 2019."},{"key":"e_1_2_1_37_1","volume-title":"SIGMOD, page 631--646","author":"Nothaft F. A.","year":"2015","unstructured":"F. A. Nothaft, M. Massie, T. Danford, Z. Zhang, U. Laserson, C. Yeksigian, J. Kottalam, A. Ahuja, J. Hammerbacher, M. Linderman, and et al. Rethinking data-intensive science using scalable analytics systems. In SIGMOD, page 631--646, New York, NY, USA, 2015. ACM."},{"key":"e_1_2_1_38_1","unstructured":"OpenStack Swift. https:\/\/www.openstack.org\/software\/releases\/train\/components\/swift."},{"key":"e_1_2_1_39_1","unstructured":"Querying external data using Amazon Redshift Spectrum. https:\/\/docs.aws.amazon.com\/redshift\/latest\/dg\/c-using-spectrum.html."},{"key":"e_1_2_1_40_1","unstructured":"S3 consistency documentation. https:\/\/docs.aws.amazon.com\/AmazonS3\/latest\/dev\/Introduction.html#ConsistencyModel."},{"key":"e_1_2_1_41_1","unstructured":"S3 ListObjectsV2 API. https:\/\/docs.aws.amazon.com\/AmazonS3\/latest\/API\/API_ListObjectsV2.html."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2019.00196"},{"key":"e_1_2_1_43_1","first-page":"553","volume-title":"Proceedings of the 31st International Conference on Very Large Data Bases, VLDB '05","author":"Stonebraker M.","year":"2005","unstructured":"M. Stonebraker, D. J. Abadi, A. Batkin, X. Chen, M. Cherniack, M. Ferreira, E. Lau, A. Lin, S. Madden, E. O'Neil, P. O'Neil, A. Rasin, N. Tran, and S. Zdonik. C-store: A column-oriented dbms. In Proceedings of the 31st International Conference on Very Large Data Bases, VLDB '05, page 553--564. VLDB Endowment, 2005."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.1001779"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2010.5447738"},{"key":"e_1_2_1_46_1","volume-title":"AWS re:Invent","author":"Tomsen Bukovec M.-L.","year":"2018","unstructured":"M.-L. Tomsen Bukovec. AWS re:Invent 2018. Building for durability in Amazon S3 and Glacier. https:\/\/www.youtube.com\/watch?v=nLyppihvhpQ, 2018."},{"key":"e_1_2_1_47_1","volume-title":"TPC benchmark DS standard specification version 2.11.0","author":"Transaction Processing Performance Council","year":"2019","unstructured":"Transaction Processing Performance Council. TPC benchmark DS standard specification version 2.11.0, 2019."},{"key":"e_1_2_1_48_1","unstructured":"Understanding block blobs append blobs and page blobs. https:\/\/docs.microsoft.com\/en-us\/rest\/api\/storageservices\/understanding-block-blobs-append-blobs-and-page-blobs."},{"key":"e_1_2_1_49_1","first-page":"1041","volume-title":"SIGMOD","author":"Verbitski A.","year":"2017","unstructured":"A. Verbitski, X. Bao, A. Gupta, D. Saha, M. Brahmadesam, K. Gupta, R. Mittal, S. Krishnamurthy, S. Maurice, and T. Kharatishvili. Amazon Aurora: Design considerations for high throughput cloud-native relational databases. In SIGMOD, pages 1041--1052, 05 2017."},{"key":"e_1_2_1_50_1","unstructured":"R. Yao and C. Crosbie. Getting started with new table formats on Dataproc. https:\/\/cloud.google.com\/blog\/products\/data-analytics\/getting-started-with-new-table-formats-on-dataproc."},{"key":"e_1_2_1_51_1","first-page":"39","article-title":"Accelerating the machine learning lifecycle with MLflow","volume":"41","author":"Zaharia M.","year":"2018","unstructured":"M. Zaharia, A. Chen, A. Davidson, A. Ghodsi, S. A. Hong, A. Konwinski, S. Murching, T. Nykodym, P. Ogilvie, M. Parkhe, F. Xie, and C. Zumar. Accelerating the machine learning lifecycle with MLflow. IEEE Data Eng. Bull., 41:39--45, 2018.","journal-title":"IEEE Data Eng. Bull."},{"key":"e_1_2_1_52_1","first-page":"15","volume-title":"NSDI","author":"Zaharia M.","year":"2012","unstructured":"M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. J. Franklin, S. Shenker, and I. Stoica. Resilient Distributed Datasets: A Fault-tolerant Abstraction for In-memory Cluster Computing. In NSDI, pages 15--28, 2012."}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3415478.3415560","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,17]],"date-time":"2025-09-17T02:44:40Z","timestamp":1758077080000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3415478.3415560"}},"subtitle":["high-performance ACID table storage over cloud object stores"],"short-title":[],"issued":{"date-parts":[[2020,8]]},"references-count":52,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2020,8]]}},"alternative-id":["10.14778\/3415478.3415560"],"URL":"https:\/\/doi.org\/10.14778\/3415478.3415560","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2020,8]]}}}