{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T15:45:27Z","timestamp":1782834327363,"version":"3.54.5"},"reference-count":43,"publisher":"Association for Computing Machinery (ACM)","issue":"12","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2023,8]]},"abstract":"<jats:p>Stream processing is widely used for real-time data processing and decision-making, leading to tens of thousands of streaming jobs deployed in ByteDance cloud. Since those streaming jobs usually run for several days or longer and the input workloads vary over time, they usually face diverse runtime issues such as processing lag and varying failures. This requires runtime management to resolve such runtime issues automatically. However, designing a runtime management service on the ByteDance scale is challenging. In particular, the service has to concurrently manage cluster-wide streaming jobs in a scalable and extensible manner. Furthermore, it should also be able to manage diverse streaming jobs effectively.<\/jats:p>\n          <jats:p>\n            To this end, we propose\n            <jats:italic toggle=\"yes\">StreamOps<\/jats:italic>\n            to enable cloud-native runtime management for streaming jobs in ByteDance.\n            <jats:italic toggle=\"yes\">StreamOps<\/jats:italic>\n            has three main designs to address the challenges. 1) To allow for scalability,\n            <jats:italic toggle=\"yes\">StreamOps<\/jats:italic>\n            is running as a standalone lightweight control plane to manage cluster-wide streaming jobs. 2) To enable extensible runtime management,\n            <jats:italic toggle=\"yes\">StreamOps<\/jats:italic>\n            abstracts control policies to identify and resolve runtime issues. New control policies can be implemented with a detect-diagnose-resolve programming paradigm. Each control policy is also configurable for different streaming jobs according to the performance requirements. 3) To mitigate processing lag and handling failures effectively,\n            <jats:italic toggle=\"yes\">StreamOps<\/jats:italic>\n            features three control policies, i.e., auto-scaler, straggler detector, and job doctor, that are inspired by state-of-the-art research and production experiences at ByteDance. In this paper, we introduce the design decisions we made and the experiences we learned from building\n            <jats:italic toggle=\"yes\">StreamOps.<\/jats:italic>\n            We evaluate\n            <jats:italic toggle=\"yes\">StreamOps<\/jats:italic>\n            in our production environment, and the experiment results have further validated our system design.\n          <\/jats:p>","DOI":"10.14778\/3611540.3611543","type":"journal-article","created":{"date-parts":[[2023,9,15]],"date-time":"2023-09-15T11:32:37Z","timestamp":1694777557000},"page":"3501-3514","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":14,"title":["StreamOps: Cloud-Native Runtime Management for Streaming Services in ByteDance"],"prefix":"10.14778","volume":"16","author":[{"given":"Yancan","family":"Mao","sequence":"first","affiliation":[{"name":"National University of Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhanghao","family":"Chen","sequence":"additional","affiliation":[{"name":"ByteDance Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yifan","family":"Zhang","sequence":"additional","affiliation":[{"name":"ByteDance Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Meng","family":"Wang","sequence":"additional","affiliation":[{"name":"ByteDance Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yong","family":"Fang","sequence":"additional","affiliation":[{"name":"ByteDance Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guanghui","family":"Zhang","sequence":"additional","affiliation":[{"name":"ByteDance Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rui","family":"Shi","sequence":"additional","affiliation":[{"name":"ByteDance Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Richard T. B.","family":"Ma","sequence":"additional","affiliation":[{"name":"National University of Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,8]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Daniel J Abadi Yanif Ahmad Magdalena Balazinska Ugur Cetintemel Mitch Cherniack Jeong-Hyon Hwang Wolfgang Lindner Anurag Maskey Alex Rasin Esther Ryvkina et al. 2005. The design of the Borealis stream processing engine.. In CIDR. 277--289."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-003-0095-z"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.14778\/2536222.2536229"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.14778\/2824032.2824076"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-014-0357-y"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2890784"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2038916.2038932"},{"key":"e_1_2_1_8_1","volume-title":"Apache flink: Stream and batch processing in a single engine. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering 36, 4","author":"Carbone Paris","year":"2015","unstructured":"Paris Carbone, Asterios Katsifodimos, Stephan Ewen, Volker Markl, Seif Haridi, and Kostas Tzoumas. 2015. Apache flink: Stream and batch processing in a single engine. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering 36, 4 (2015). http:\/\/flink.apache.org\/"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2463676.2465282"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2904441"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3318464.3389723"},{"key":"e_1_2_1_12_1","volume-title":"Cloud computing patterns: fundamentals to design, build, and manage cloud applications","author":"Fehling Christoph","unstructured":"Christoph Fehling, Frank Leymann, Ralph Retter, Walter Schupeck, and Peter Arbitter. 2014. Cloud computing patterns: fundamentals to design, build, and manage cloud applications. Vol. 545. Springer."},{"key":"e_1_2_1_13_1","volume-title":"Flink Forward Asia","author":"Flink Apache","year":"2021","unstructured":"Apache Flink. (2021). Flink Forward Asia 2021. https:\/\/flink-forward.org.cn\/"},{"key":"e_1_2_1_14_1","volume-title":"AutoScaling model proposal in Apache Flink. https:\/\/cwiki.apache.org\/confluence\/display\/FLINK\/FLIP-271%3A+Autoscaling","author":"Flink Apache","year":"2022","unstructured":"Apache Flink. (2022). AutoScaling model proposal in Apache Flink. https:\/\/cwiki.apache.org\/confluence\/display\/FLINK\/FLIP-271%3A+Autoscaling"},{"key":"e_1_2_1_15_1","volume-title":"Elastic scaling execution in Apache Flink. https:\/\/nightlies.apache.org\/flink\/flink-docs-master\/docs\/deployment\/elastic_scaling\/","author":"Flink Apache","year":"2022","unstructured":"Apache Flink. (2022). Elastic scaling execution in Apache Flink. https:\/\/nightlies.apache.org\/flink\/flink-docs-master\/docs\/deployment\/elastic_scaling\/"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.14778\/3137765.3137786"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDCS.2015.49"},{"key":"e_1_2_1_18_1","volume-title":"13th USENIX Symposium on Operating Systems Design and Implementation (OSDI). 213--231","author":"Gjengset Jon","year":"2018","unstructured":"Jon Gjengset, Malte Schwarzkopf, Jonathan Behrens, Lara Timb\u00f3 Ara\u00fajo, Martin Ek, Eddie Kohler, M. Frans Kaashoek, and Robert Morris. 2018. Noria: dynamic, partially-stateful data-flow for high-performance web applications. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI). 213--231."},{"key":"e_1_2_1_19_1","volume-title":"2022 USENIX Annual Technical Conference (USENIX ATC 22)","author":"Gu Rong","year":"2022","unstructured":"Rong Gu, Han Yin, Weichang Zhong, Chunfeng Yuan, and Yihua Huang. 2022. Meces: Latency-efficient Rescaling via Prioritized State Migration for Stateful Distributed Stream Processing Systems. In 2022 USENIX Annual Technical Conference (USENIX ATC 22). 539--556."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.14778\/3329772.3329777"},{"key":"e_1_2_1_21_1","volume-title":"Flavio Paiva Junqueira, and Benjamin Reed","author":"Hunt Patrick","year":"2010","unstructured":"Patrick Hunt, Mahadev Konar, Flavio Paiva Junqueira, and Benjamin Reed. 2010. ZooKeeper: wait-free coordination for internet-scale systems.. In USENIX annual technical conference, Vol. 8."},{"key":"e_1_2_1_22_1","doi-asserted-by":"crossref","unstructured":"Michael Isard Mihai Budiu Yuan Yu Andrew Birrell and Dennis Fetterly. 2007. Dryad: distributed data-parallel programs from sequential building blocks. In ACM SIGOPS operating systems review. ACM 59--72.","DOI":"10.1145\/1272998.1273005"},{"key":"e_1_2_1_23_1","volume-title":"13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18)","author":"Kalavri Vasiliki","year":"2018","unstructured":"Vasiliki Kalavri, John Liagouris, Moritz Hoffmann, Desislava Dimitrova, Matthew Forshaw, and Timothy Roscoe. 2018. Three steps is all you need: fast, accurate, automatic scaling decisions for distributed streaming dataflows. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). 783--798."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3267809.3267832"},{"key":"e_1_2_1_25_1","volume-title":"Proceedings of the NetDB","volume":"11","author":"Kreps Jay","year":"2011","unstructured":"Jay Kreps, Neha Narkhede, Jun Rao, et al. 2011. Kafka: A distributed messaging system for log processing. In Proceedings of the NetDB, Vol. 11. Athens, Greece, 1--7."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/360051.360074"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3477132.3483546"},{"key":"e_1_2_1_28_1","volume-title":"13th USENIX Symposium on Networked Systems Design and Implementation (NSDI 16)","author":"Lin Wei","year":"2016","unstructured":"Wei Lin, Zhengping Qian, Junwei Xu, Sen Yang, Jingren Zhou, and Lidong Zhou. 2016. Streamscope: continuous reliable distributed processing of big data streams. In 13th USENIX Symposium on Networked Systems Design and Implementation (NSDI 16). 439--453."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDCS.2015.48"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.14778\/3231751.3231765"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3472883.3487010"},{"key":"e_1_2_1_32_1","unstructured":"Yancan Mao Jianjun Zhao Shuhao Zhang Haikun Liu and Volker Markl. 2022. MorphStream: Adaptive Scheduling for Scalable Transactional Stream Processing on Multicores. (2022)."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE48307.2020.00141"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2517349.2522738"},{"key":"e_1_2_1_35_1","volume-title":"Proc. 8th ACM\/USENIX Symposium on Networked Systems Design and Implementation. 113--126","author":"Murray Derek G","year":"2011","unstructured":"Derek G Murray, Malte Schwarzkopf, Christopher Smowton, Steven Smith, Anil Madhavapeddy, and Steven Hand. 2011. CIEL: a universal execution engine for distributed data-flow computing. In Proc. 8th ACM\/USENIX Symposium on Networked Systems Design and Implementation. 113--126."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2465351.2465353"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1002\/widm.2"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3448016.3457320"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.14778\/1687553.1687609"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3448016.3457556"},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation. USENIX Association, 2--2.","author":"Zaharia Matei","year":"2012","unstructured":"Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J Franklin, Scott Shenker, and Ion Stoica. 2012. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation. USENIX Association, 2--2."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3300067"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132777"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3611540.3611543","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,10]],"date-time":"2025-09-10T22:37:07Z","timestamp":1757543827000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3611540.3611543"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8]]},"references-count":43,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2023,8]]}},"alternative-id":["10.14778\/3611540.3611543"],"URL":"https:\/\/doi.org\/10.14778\/3611540.3611543","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2023,8]]},"assertion":[{"value":"2023-08-01","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}