{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,12]],"date-time":"2026-05-12T16:58:29Z","timestamp":1778605109828,"version":"3.51.4"},"reference-count":47,"publisher":"Association for Computing Machinery (ACM)","issue":"12","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2017,8]]},"abstract":"<jats:p>Stream processors are emerging in industry as an apparatus that drives analytical but also mission critical services handling the core of persistent application logic. Thus, apart from scalability and low-latency, a rising system need is first-class support for application state together with strong consistency guarantees, and adaptivity to cluster reconfigurations, software patches and partial failures. Although prior systems research has addressed some of these specific problems, the practical challenge lies on how such guarantees can be materialized in a transparent, non-intrusive manner that relieves the user from unnecessary constraints. Such needs served as the main design principles of state management in Apache Flink, an open source, scalable stream processor.<\/jats:p>\n          <jats:p>We present Flink's core pipelined, in-flight mechanism which guarantees the creation of lightweight, consistent, distributed snapshots of application state, progressively, without impacting continuous execution. Consistent snapshots cover all needs for system reconfiguration, fault tolerance and version management through coarse grained rollback recovery. Application state is declared explicitly to the system, allowing efficient partitioning and transparent commits to persistent storage. We further present Flink's backend implementations and mechanisms for high availability, external state queries and output commit. Finally, we demonstrate how these mechanisms behave in practice with metrics and large-deployment insights exhibiting the low performance trade-offs of our approach and the general benefits of exploiting asynchrony in continuous, yet sustainable system deployments.<\/jats:p>","DOI":"10.14778\/3137765.3137777","type":"journal-article","created":{"date-parts":[[2017,9,7]],"date-time":"2017-09-07T13:35:53Z","timestamp":1504791353000},"page":"1718-1729","source":"Crossref","is-referenced-by-count":203,"title":["State management in Apache Flink\u00ae"],"prefix":"10.14778","volume":"10","author":[{"given":"Paris","family":"Carbone","sequence":"first","affiliation":[{"name":"KTH Royal Institute of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stephan","family":"Ewen","sequence":"additional","affiliation":[{"name":"data Artisans"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gyula","family":"F\u00f3ra","sequence":"additional","affiliation":[{"name":"King Digital Entertainment Limited"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Seif","family":"Haridi","sequence":"additional","affiliation":[{"name":"KTH Royal Institute of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stefan","family":"Richter","sequence":"additional","affiliation":[{"name":"data Artisans"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kostas","family":"Tzoumas","sequence":"additional","affiliation":[{"name":"data Artisans"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2017,8]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"AthenaX : Ubers stream processing platform on Flink. http:\/\/sf.flink-forward.org\/kb_sessions\/athenax-ubers-streaming-processing-platform-on-flink\/.  AthenaX : Ubers stream processing platform on Flink. http:\/\/sf.flink-forward.org\/kb_sessions\/athenax-ubers-streaming-processing-platform-on-flink\/."},{"key":"e_1_2_1_2_1","unstructured":"Blink: How Alibaba Uses Apache Flink. http:\/\/data-artisans.com\/blink-flink-alibaba-search\/ 2016.  Blink: How Alibaba Uses Apache Flink. http:\/\/data-artisans.com\/blink-flink-alibaba-search\/ 2016."},{"key":"e_1_2_1_3_1","volume-title":"https:\/\/www.oreilly.com\/learning\/apache-spark-2-0--introduction-to-structured-streaming","author":"Spark's Structured Streaming Introduction","year":"2016"},{"key":"e_1_2_1_4_1","volume-title":"Scalable Real-Time Analytics at King. https:\/\/techblog.king.com\/rbea-scalable-real-time-analytics-king\/","author":"Rbea","year":"2016"},{"key":"e_1_2_1_5_1","volume-title":"https:\/\/apex.apache.org","author":"Apex Apache","year":"2017"},{"key":"e_1_2_1_6_1","volume-title":"https:\/\/beam.apache.org\/","author":"Beam Apache","year":"2017"},{"key":"e_1_2_1_7_1","volume-title":"http:\/\/flink.apache.org\/","author":"Flink Apache","year":"2017"},{"key":"e_1_2_1_8_1","volume-title":"http:\/\/storm.apache.org\/","author":"Storm Apache","year":"2017"},{"key":"e_1_2_1_9_1","volume-title":"http:\/\/flink-forward.org\/","author":"Forward Flink","year":"2017"},{"key":"e_1_2_1_10_1","unstructured":"Flink Survey. http:\/\/data-artisans.com\/flink-user-survey-2016-part-1\/ http:\/\/data-artisans.com\/flink-user-survey-2016-part-2\/ 2017.  Flink Survey. http:\/\/data-artisans.com\/flink-user-survey-2016-part-1\/ http:\/\/data-artisans.com\/flink-user-survey-2016-part-2\/ 2017."},{"key":"e_1_2_1_11_1","volume-title":"https:\/\/cloud.google.com\/dataflow\/","author":"Dataflow Google Cloud","year":"2017"},{"key":"e_1_2_1_12_1","volume-title":"http:\/\/2016.flink-forward.org\/kb_sessions\/a-brief-history-of-time-with-apache-flink-real-time-monitoring-and-analysis-with-flink-kafka-hb\/","author":"Kafka","year":"2017"},{"key":"e_1_2_1_13_1","volume-title":"http:\/\/rocksdb.org\/","year":"2017"},{"key":"e_1_2_1_14_1","volume-title":"http:\/\/sf.flink-forward.org\/kb_sessions\/keynote-stream-processing-with-flink-at-netflix\/","author":"Netflix Stream","year":"2017"},{"key":"e_1_2_1_15_1","unstructured":"StreamING models how ING adds models at runtime to catch fraudsters. http:\/\/sf.flink-forward.org\/kb_sessions\/streaming-models-how-ing-adds-models-at-runtime-to-catch-fraudsters\/ 2017.  StreamING models how ING adds models at runtime to catch fraudsters. http:\/\/sf.flink-forward.org\/kb_sessions\/streaming-models-how-ing-adds-models-at-runtime-to-catch-fraudsters\/ 2017."},{"key":"e_1_2_1_16_1","volume-title":"http:\/\/storm.apache.org\/releases\/0.10.0\/Trident-tutorial.html","author":"Stream Processing Programming Model The Trident","year":"2017"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-003-0095-z"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.14778\/2536222.2536229"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.14778\/2824032.2824076"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-014-0357-y"},{"key":"e_1_2_1_21_1","volume-title":"Stream: The stanford data stream management system. Book chapter","author":"Arasu A.","year":"2004"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807128.1807148"},{"key":"e_1_2_1_23_1","first-page":"28","author":"Carbone P.","year":"2015","journal-title":"IEEE Data Engineering Bulletin, page"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2463676.2465282"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1809028.1806638"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/872757.872857"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/214451.214456"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1365815.1365816"},{"key":"e_1_2_1_29_1","first-page":"257","volume":"3","author":"Cherniack M.","year":"2003","journal-title":"Scalable distributed stream processing. In CIDR"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.5555\/2789272.2789277"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/1327452.1327492"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/1323293.1294281"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/568522.568525"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807128.1807139"},{"key":"e_1_2_1_35_1","volume-title":"NSDI","author":"Hindman B.","year":"2011"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2528412"},{"key":"e_1_2_1_37_1","first-page":"9","volume-title":"USENIX annual technical conference","author":"Hunt P.","year":"2010"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.14778\/3007263.3007272"},{"key":"e_1_2_1_39_1","volume-title":"NetDB","author":"Kreps J.","year":"2011"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.14778\/2212351.2212354"},{"key":"e_1_2_1_41_1","volume-title":"Big Data: Principles and best practices of scalable realtime data systems","author":"Marz N.","year":"2015"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2517349.2522738"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2523616.2523633"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2012.108"},{"key":"e_1_2_1_45_1","unstructured":"Y. Yu M. Isard D. Fetterly M. Budiu \u00da. Erlingsson P. K. Gunda and J. Currey. Dryadlinq: A system for general-purpose distributed data-parallel computing using a high-level language.  Y. Yu M. Isard D. Fetterly M. Budiu \u00da. Erlingsson P. K. Gunda and J. Currey. Dryadlinq: A system for general-purpose distributed data-parallel computing using a high-level language."},{"key":"e_1_2_1_46_1","volume-title":"HotCloud","author":"Zaharia M.","year":"2010"},{"key":"e_1_2_1_47_1","first-page":"10","volume-title":"Proceedings of the 4th USENIX conference on Hot Topics in Cloud Ccomputing","author":"Zaharia M.","year":"2012"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3137765.3137777","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T10:12:49Z","timestamp":1672222369000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3137765.3137777"}},"subtitle":["consistent stateful distributed stream processing"],"short-title":[],"issued":{"date-parts":[[2017,8]]},"references-count":47,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2017,8]]}},"alternative-id":["10.14778\/3137765.3137777"],"URL":"https:\/\/doi.org\/10.14778\/3137765.3137777","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2017,8]]}}}