{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,26]],"date-time":"2025-11-26T16:38:15Z","timestamp":1764175095214,"version":"3.41.0"},"reference-count":39,"publisher":"Association for Computing Machinery (ACM)","issue":"5s","license":[{"start":{"date-parts":[[2021,9,22]],"date-time":"2021-09-22T00:00:00Z","timestamp":1632268800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"NSF","award":["1646579 (CPS)"],"award-info":[{"award-number":["1646579 (CPS)"]}]},{"DOI":"10.13039\/501100019814","name":"Fullgraf Foundation","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100019814","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Embed. Comput. Syst."],"published-print":{"date-parts":[[2021,10,31]]},"abstract":"<jats:p>Fault-tolerant coordination services have been widely used in distributed applications in cloud environments. Recent years have witnessed the emergence of time-sensitive applications deployed in edge computing environments, which introduces both challenges and opportunities for coordination services. On one hand, coordination services must recover from failures in a timely manner. On the other hand, edge computing employs local networked platforms that can be exploited to achieve timely recovery. In this work, we first identify the limitations of the leader election and recovery protocols underlying Apache ZooKeeper, the prevailing open-source coordination service. To reduce recovery latency from leader failures, we then design RT-Zookeeper with a set of novel features including a fast-convergence election protocol, a quorum channel notification mechanism, and a distributed epoch persistence protocol. We have implemented RT-Zookeeper based on ZooKeeper version 3.5.8. Empirical evaluation shows that RT-ZooKeeper achieves 91% reduction in maximum recovery latency in comparison to ZooKeeper. Furthermore, a case study demonstrates that fast failure recovery in RT-ZooKeeper can benefit a common messaging service like Kafka in terms of message latency.<\/jats:p>","DOI":"10.1145\/3477034","type":"journal-article","created":{"date-parts":[[2021,9,22]],"date-time":"2021-09-22T20:48:40Z","timestamp":1632343720000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":16,"title":["RT-ZooKeeper: Taming the Recovery Latency of a Coordination Service"],"prefix":"10.1145","volume":"20","author":[{"given":"Haoran","family":"Li","sequence":"first","affiliation":[{"name":"Cyber-Physical SystemsLaboratory, Washington University in St. Louis, St. Louis, MO, USA"}]},{"given":"Chenyang","family":"Lu","sequence":"additional","affiliation":[{"name":"Cyber-Physical SystemsLaboratory, Washington University in St. Louis, St. Louis, MO, USA"}]},{"given":"Christopher D.","family":"Gill","sequence":"additional","affiliation":[{"name":"Cyber-Physical SystemsLaboratory, Washington University in St. Louis, St. Louis, MO, USA"}]}],"member":"320","published-online":{"date-parts":[[2021,9,22]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/RTAS.2010.30"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/RTAS.2009.36"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/SRDS.2012.66"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.5555\/1972457.1972472"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.5555\/1298455.1298487"},{"key":"e_1_2_1_6_1","first-page":"1","article-title":"Apache flink: Stream and batch processing in a single engine","volume":"36","author":"Carbone Paris","year":"2015","unstructured":"Paris Carbone , Asterios Katsifodimos , Stephan Ewen , Volker Markl , Seif Haridi , and Kostas Tzoumas . 2015 . Apache flink: Stream and batch processing in a single engine . Bulletin of the IEEE Computer Society Technical Committee on Data Engineering 36 , 4 (2015), 1 \u2013 1 . Paris Carbone, Asterios Katsifodimos, Stephan Ewen, Volker Markl, Seif Haridi, and Kostas Tzoumas. 2015. Apache flink: Stream and batch processing in a single engine. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering 36, 4 (2015), 1\u20131.","journal-title":"Bulletin of the IEEE Computer Society Technical Committee on Data Engineering"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/RTAS52030.2021.00038"},{"key":"e_1_2_1_8_1","unstructured":"Dell. Inc.2020. Edge Analytics for Industry 4.0 with Confluent Platform. https:\/\/www.delltechnologies.com\/asset\/en-us\/products\/ready-solutions\/technical-support\/h18352-da-edge-iiot-confluent-dellencinfra-ra.pdf. Accessed: 2021-06-04. Dell. Inc.2020. Edge Analytics for Industry 4.0 with Confluent Platform. https:\/\/www.delltechnologies.com\/asset\/en-us\/products\/ready-solutions\/technical-support\/h18352-da-edge-iiot-confluent-dellencinfra-ra.pdf. Accessed: 2021-06-04."},{"key":"e_1_2_1_9_1","first-page":"1","article-title":"Improving zookeeper atomic broadcast performance when a server quorum never crashes","volume":"5","author":"Sanosi Ibrahim","year":"2018","unstructured":"Ibrahim EL- Sanosi and Paul Ezhilchelvan . 2018 . Improving zookeeper atomic broadcast performance when a server quorum never crashes . EAI Endorsed Transactions on Energy Web 5 , 17 (2018), 1 \u2013 1 . Ibrahim EL-Sanosi and Paul Ezhilchelvan. 2018. Improving zookeeper atomic broadcast performance when a server quorum never crashes. EAI Endorsed Transactions on Energy Web 5, 17 (2018), 1\u20131.","journal-title":"EAI Endorsed Transactions on Energy Web"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/RTAS48715.2020.00-13"},{"volume-title":"2020 IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS). IEEE","author":"Gujarati Arpan","key":"e_1_2_1_11_1","unstructured":"Arpan Gujarati , Sergey Bozhko , and Bj\u00f6rn B. Brandenburg . 2020. Real-time replica consistency over ethernet with reliability bounds . In 2020 IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS). IEEE , Sydney, 376\u2013389. Arpan Gujarati, Sergey Bozhko, and Bj\u00f6rn B. Brandenburg. 2020. Real-time replica consistency over ethernet with reliability bounds. In 2020 IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS). IEEE, Sydney, 376\u2013389."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.5555\/1855840.1855851"},{"key":"e_1_2_1_13_1","volume-title":"European Workshop on Performance Engineering. Springer","author":"Ibrahim Sanosi","year":"2017","unstructured":"EL- Sanosi Ibrahim and Paul Ezhilchelvan . 2017 . Improving zookeeper atomic broadcast performance by coin tossing . In European Workshop on Performance Engineering. Springer , Berlin, 249\u2013265. EL-Sanosi Ibrahim and Paul Ezhilchelvan. 2017. Improving zookeeper atomic broadcast performance by coin tossing. In European Workshop on Performance Engineering. Springer, Berlin, 249\u2013265."},{"key":"e_1_2_1_14_1","unstructured":"Flavio Junqueira. 2010. Last processed zxid set prematurely while establishing leadership. https:\/\/issues.apache.org\/jira\/browse\/ZOOKEEPER-790. Accessed: 2020-10-01. Flavio Junqueira. 2010. Last processed zxid set prematurely while establishing leadership. https:\/\/issues.apache.org\/jira\/browse\/ZOOKEEPER-790. Accessed: 2020-10-01."},{"key":"e_1_2_1_15_1","unstructured":"Flavio Junqueira. 2013. Zab Pre 1.0. https:\/\/cwiki.apache.org\/confluence\/display\/ZOOKEEPER\/Zab+Pre+1.0. Accessed: 2020-10-01. Flavio Junqueira. 2013. Zab Pre 1.0. https:\/\/cwiki.apache.org\/confluence\/display\/ZOOKEEPER\/Zab+Pre+1.0. Accessed: 2020-10-01."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSN.2011.5958223"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-35668-1_31"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2968478.2968480"},{"key":"e_1_2_1_19_1","doi-asserted-by":"crossref","unstructured":"Mahadev Konar. 2011. Zookeeper servers should commit the new leader txn to their logs. https:\/\/issues.apache.org\/jira\/browse\/ZOOKEEPER-335. Accessed: 2020-10-01. Mahadev Konar. 2011. Zookeeper servers should commit the new leader txn to their logs. https:\/\/issues.apache.org\/jira\/browse\/ZOOKEEPER-335. Accessed: 2020-10-01.","DOI":"10.18844\/gjpr.v10i1.4116"},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the NetDB","volume":"11","author":"Kreps Jay","year":"2011","unstructured":"Jay Kreps , Neha Narkhede , Jun Rao , et\u00a0al. 2011 . Kafka: A distributed messaging system for log processing . In Proceedings of the NetDB , Vol. 11 . IEEE, Athens, 1\u20137. Jay Kreps, Neha Narkhede, Jun Rao, et\u00a0al. 2011. Kafka: A distributed messaging system for log processing. In Proceedings of the NetDB, Vol. 11. IEEE, Athens, 1\u20137."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/279227.279229"},{"key":"e_1_2_1_22_1","first-page":"18","article-title":"Paxos made simple","volume":"32","author":"\u00a0al Leslie Lamport","year":"2001","unstructured":"Leslie Lamport et \u00a0al . 2001 . Paxos made simple . ACM Sigact News 32 , 4 (2001), 18 \u2013 25 . Leslie Lamport et\u00a0al. 2001. Paxos made simple. ACM Sigact News 32, 4 (2001), 18\u201325.","journal-title":"ACM Sigact News"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/RTAS52030.2021.00036"},{"key":"e_1_2_1_24_1","unstructured":"Lucidworks. 2018. Setting Up an External ZooKeeper Ensemble. https:\/\/doc.lucidworks.com\/fusion-server\/4.2\/reference\/solr-reference-guide\/7.5.0\/setting-up-an-external-zookeeper-ensemble.html. Accessed: 2020-10-01. Lucidworks. 2018. Setting Up an External ZooKeeper Ensemble. https:\/\/doc.lucidworks.com\/fusion-server\/4.2\/reference\/solr-reference-guide\/7.5.0\/setting-up-an-external-zookeeper-ensemble.html. Accessed: 2020-10-01."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.5555\/525656"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.5555\/525656"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.5555\/525656"},{"key":"e_1_2_1_29_1","unstructured":"Benjamin Reed. 2016. Zab 1.0. https:\/\/cwiki.apache.org\/confluence\/display\/ZOOKEEPER\/Zab1.0. Accessed: 2020-10-01. Benjamin Reed. 2016. Zab 1.0. https:\/\/cwiki.apache.org\/confluence\/display\/ZOOKEEPER\/Zab1.0. Accessed: 2020-10-01."},{"key":"e_1_2_1_30_1","unstructured":"Benjamin Reed and Norbert Kalmar. 2019. Applications Powered by ZooKeeper. https:\/\/cwiki.apache.org\/confluence\/display\/ZOOKEEPER\/PoweredBy. Benjamin Reed and Norbert Kalmar. 2019. Applications Powered by ZooKeeper. https:\/\/cwiki.apache.org\/confluence\/display\/ZOOKEEPER\/PoweredBy."},{"volume-title":"Do not overpay for fault tolerance! In IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS\u201921)","author":"Roth Edo","key":"e_1_2_1_31_1","unstructured":"Edo Roth and Andreas Haeberlen . 2021. Do not overpay for fault tolerance! In IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS\u201921) . IEEE , Nashville , 151\u2013160. Edo Roth and Andreas Haeberlen. 2021. Do not overpay for fault tolerance! In IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS\u201921). IEEE, Nashville, 151\u2013160."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/PRDC.2011.19"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSST.2010.5496972"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/RTSS.2013.11"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.14778\/1687553.1687609"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2595641"},{"key":"e_1_2_1_37_1","unstructured":"Elin Vinka. 2018. How many Zookeepers in a cluster?https:\/\/www.cloudkarafka.com\/blog\/2018-07-04-cloudkarafka-how-many-zookeepers-in-a-cluster.html. Accessed: 2020-10-01. Elin Vinka. 2018. How many Zookeepers in a cluster?https:\/\/www.cloudkarafka.com\/blog\/2018-07-04-cloudkarafka-how-many-zookeepers-in-a-cluster.html. Accessed: 2020-10-01."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDCS.2019.00101"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2656045.2656066"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/RTAS.2017.15"}],"container-title":["ACM Transactions on Embedded Computing Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3477034","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3477034","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3477034","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:30:46Z","timestamp":1750188646000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3477034"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,22]]},"references-count":39,"journal-issue":{"issue":"5s","published-print":{"date-parts":[[2021,10,31]]}},"alternative-id":["10.1145\/3477034"],"URL":"https:\/\/doi.org\/10.1145\/3477034","relation":{},"ISSN":["1539-9087","1558-3465"],"issn-type":[{"type":"print","value":"1539-9087"},{"type":"electronic","value":"1558-3465"}],"subject":[],"published":{"date-parts":[[2021,9,22]]},"assertion":[{"value":"2021-04-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-07-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-09-22","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}