{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,1]],"date-time":"2026-06-01T20:34:49Z","timestamp":1780346089788,"version":"3.54.1"},"reference-count":27,"publisher":"Association for Computing Machinery (ACM)","issue":"12","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2025,8]]},"abstract":"<jats:p>Data lakehouse architectures unify the cost-efficiency of data lakes with the transactional guarantees of data warehouses. Yet, real-time ingestion often depends on external streaming systems such as Apache Kafka, along with bespoke connectors that read from Kafka and write into the lakehouse\u2014leading to increased complexity and high operational costs. In particular, traditional leader-based data streaming platforms are designed for sub-100 ms low-latency workloads; however, when used for data-intensive ingestion in a cloud environment, cross availability-zone (AZ) disk-based replication significantly raises total infrastructure costs due to excessive network traffic and overprovisioned disk storage. This paper introduces Ursa, a leaderless, cloud-native, and Kafka-compatible streaming engine that writes data directly to open lakehouse tables on object storage. By eliminating leader-based replication, disk-based broker storage, and external connectors, Ursa markedly reduces infrastructure costs while preserving high throughput, exactly-once semantics, and near-real-time streaming capabilities. Experimental results show that Ursa matches the performance of traditional Kafka clusters at a fraction of the cost, offering up to a 10x reduction in infrastructure expenses.<\/jats:p>","DOI":"10.14778\/3750601.3750636","type":"journal-article","created":{"date-parts":[[2025,9,16]],"date-time":"2025-09-16T13:38:05Z","timestamp":1758029885000},"page":"5184-5196","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Ursa: A Lakehouse-Native Data Streaming Engine for Kafka"],"prefix":"10.14778","volume":"18","author":[{"given":"Matteo","family":"Merli","sequence":"first","affiliation":[{"name":"StreamNative, Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sijie","family":"Guo","sequence":"additional","affiliation":[{"name":"StreamNative, Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Penghui","family":"Li","sequence":"additional","affiliation":[{"name":"StreamNative, Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hang","family":"Chen","sequence":"additional","affiliation":[{"name":"StreamNative, Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Neng","family":"Lu","sequence":"additional","affiliation":[{"name":"StreamNative, Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,9,16]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.14778\/3415478.3415560"},{"key":"e_1_2_1_2_1","volume-title":"JR: A High-Performance Data Generation and Benchmarking Tool. Retrieved","author":"Tool Authors JR","year":"2025","unstructured":"JR Tool Authors. 2025. JR: A High-Performance Data Generation and Benchmarking Tool. Retrieved March 1, 2025 from https:\/\/github.com\/jr-tool"},{"key":"e_1_2_1_3_1","volume-title":"Redpanda Cloud brings the fastest Kafka\u00ae API to the cloud. Retrieved","author":"Balakrishnan Praseed","year":"2025","unstructured":"Praseed Balakrishnan. 2022. Redpanda Cloud brings the fastest Kafka\u00ae API to the cloud. Retrieved March 1, 2025 from https:\/\/redpanda.com\/blog\/introducing-redpanda-cloud-for-kafka"},{"key":"e_1_2_1_4_1","volume-title":"Exactly-Once Semantics in Apache Kafka. Retrieved","year":"2025","unstructured":"Confluent. 2017. Exactly-Once Semantics in Apache Kafka. Retrieved March 1, 2025 from https:\/\/www.confluent.io\/blog\/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it\/"},{"key":"e_1_2_1_5_1","volume-title":"Freight Clusters: High-Capacity Kafka Deployments. Retrieved","year":"2025","unstructured":"Confluent. 2025. Freight Clusters: High-Capacity Kafka Deployments. Retrieved March 1, 2025 from https:\/\/www.confluent.io\/product\/freight-cluster\/"},{"key":"e_1_2_1_6_1","volume-title":"Zero-Disk Apache Kafka Alternative. Retrieved","year":"2025","unstructured":"Confluent. 2025. WarpStream: A Cloud-Native, Zero-Disk Apache Kafka Alternative. Retrieved March 1, 2025 from https:\/\/www.warpstream.com"},{"key":"e_1_2_1_7_1","volume-title":"Kafka Connect Fundamentals: What is Kafka Connect? Retrieved","author":"Inc. Confluent. 2025.","year":"2025","unstructured":"Inc. Confluent. 2025. Kafka Connect Fundamentals: What is Kafka Connect? Retrieved March 1, 2025 from https:\/\/www.confluent.io\/blog\/kafka-connect-tutorial\/"},{"key":"e_1_2_1_8_1","volume-title":"Databricks Unity Catalog. Retrieved","year":"2025","unstructured":"Databricks. 2025. Databricks Unity Catalog. Retrieved March 1, 2025 from https:\/\/www.databricks.com\/product\/unity-catalog"},{"key":"e_1_2_1_9_1","volume-title":"Delta Lake UniForm: Unified Table Formats for Interoperability in the Lakehouse. Retrieved","year":"2025","unstructured":"Databricks. 2025. Delta Lake UniForm: Unified Table Formats for Interoperability in the Lakehouse. Retrieved March 1, 2025 from https:\/\/www.databricks.com\/blog\/2023\/05\/16\/introducing-delta-lake-uniform-way-unify-your-data.html"},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of the 6th Symposium on Operating Systems Design and Implementation (OSDI'04)","author":"Dean Jeffrey","year":"2004","unstructured":"Jeffrey Dean and Sanjay Ghemawat. 2004. MapReduce: Simplified Data Processing on Large Clusters. In Proceedings of the 6th Symposium on Operating Systems Design and Implementation (OSDI'04). 137\u2013150. https:\/\/www.usenix.org\/legacy\/events\/osdi04\/tech\/dean.html"},{"key":"e_1_2_1_11_1","volume-title":"Flink Dynamic Tables. Retrieved","author":"Flink Apache","year":"2025","unstructured":"Apache Flink. 2025. Flink Dynamic Tables. Retrieved Jun 9, 2025 from https:\/\/nightlies.apache.org\/flink\/flink-docs-master\/docs\/dev\/table\/concepts\/dynamic_tables\/"},{"key":"e_1_2_1_12_1","unstructured":"Apache Software Foundation. 2025. Apache BookKeeper. Retrieved March 1 2025 from https:\/\/bookkeeper.apache.org\/"},{"key":"e_1_2_1_13_1","unstructured":"Apache Software Foundation. 2025. Apache Flink. Retrieved March 1 2025 from https:\/\/flink.apache.org\/"},{"key":"e_1_2_1_14_1","unstructured":"Apache Software Foundation. 2025. Apache Iceberg. Retrieved March 1 2025 from https:\/\/iceberg.apache.org\/"},{"key":"e_1_2_1_15_1","unstructured":"Apache Software Foundation. 2025. Apache Kafka. Retrieved March 1 2025 from https:\/\/kafka.apache.org\/"},{"key":"e_1_2_1_16_1","unstructured":"Apache Software Foundation. 2025. Apache Parquet. Retrieved March 1 2025 from https:\/\/parquet.apache.org\/"},{"key":"e_1_2_1_17_1","unstructured":"Apache Software Foundation. 2025. Apache Pulsar. Retrieved March 1 2025 from https:\/\/pulsar.apache.org\/"},{"key":"e_1_2_1_18_1","unstructured":"Apache Software Foundation. 2025. Apache Spark. Retrieved March 1 2025 from https:\/\/spark.apache.org\/"},{"key":"e_1_2_1_19_1","volume-title":"Universal Table Formats Interoperability. Retrieved","author":"Foundation Apache Software","year":"2025","unstructured":"Apache Software Foundation. 2025. Apache XTable (Incubating): Universal Table Formats Interoperability. Retrieved March 1, 2025 from https:\/\/incubator.apache.org\/projects\/xtable.html"},{"key":"e_1_2_1_20_1","unstructured":"Trino Software Foundation. 2025. Trino. Retrieved March 1 2025 from https:\/\/trino.io\/"},{"key":"e_1_2_1_21_1","volume-title":"A Guide to Evaluating the Infrastructure Costs of Apache Pulsar and Apache Kafka. Retrieved","author":"Guo Sijie","year":"2025","unstructured":"Sijie Guo. 2024. A Guide to Evaluating the Infrastructure Costs of Apache Pulsar and Apache Kafka. Retrieved March 1, 2025 from https:\/\/streamnative.io\/blog\/a-guide-to-evaluating-the-infrastructure-costs-of-apache-pulsar-and-apache-kafka"},{"key":"e_1_2_1_22_1","volume-title":"Snowflake Open Catalog. Retrieved","author":"Snowflake Inc. 2025.","year":"2025","unstructured":"Snowflake Inc. 2025. Snowflake Open Catalog. Retrieved March 1, 2025 from https:\/\/www.snowflake.com\/en\/product\/features\/open-catalog\/"},{"key":"e_1_2_1_23_1","first-page":"4","article-title":"Paxos Made Simple","volume":"32","author":"Lamport Leslie","year":"2001","unstructured":"Leslie Lamport. 2001. Paxos Made Simple. ACM SIGACT News 32, 4 (December 2001), 51\u201358. https:\/\/lamport.azurewebsites.net\/pubs\/paxos-simple.pdf","journal-title":"ACM SIGACT News"},{"key":"e_1_2_1_24_1","volume-title":"Introducing Oxia: Scalable Metadata and Coordination. Retrieved","author":"Merli Matteo","year":"2023","unstructured":"Matteo Merli. 2023. Introducing Oxia: Scalable Metadata and Coordination. Retrieved March 1, 2025 from https:\/\/streamnative.io\/blog\/introducing-oxia-scalable-metadata-and-coordination"},{"key":"e_1_2_1_25_1","volume-title":"Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics. In CIDR","author":"Armbrust Michael","year":"2021","unstructured":"et al. Michael Armbrust, Ali Ghodsi. 2021. Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics. In CIDR. http:\/\/cidrdb.org\/cidr2021\/papers\/cidr2021_paper17.pdf 11th Conference on Innovative Data Systems Research (CIDR), January 2021, Virtual."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.5555\/2643634.2643666"},{"key":"e_1_2_1_27_1","unstructured":"Apache Paimon. 2025. Apache Paimon. Retrieved Jun 9 2025 from https:\/\/paimon.apache.org\/"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3750601.3750636","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,16]],"date-time":"2025-09-16T13:38:38Z","timestamp":1758029918000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3750601.3750636"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8]]},"references-count":27,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2025,8]]}},"alternative-id":["10.14778\/3750601.3750636"],"URL":"https:\/\/doi.org\/10.14778\/3750601.3750636","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2025,8]]},"assertion":[{"value":"2025-09-16","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}