{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,13]],"date-time":"2026-06-13T04:58:24Z","timestamp":1781326704614,"version":"3.54.1"},"reference-count":59,"publisher":"Association for Computing Machinery (ACM)","issue":"6","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Manag. Data"],"published-print":{"date-parts":[[2025,12,4]]},"abstract":"<jats:p>Effective system resource management is key to delivering high performance stream processing. Stream processing engines (SPEs) rely on their host operating system (OS) for managing compute and memory resources, but this is inefficient as the OS is not stream-aware, i.e., the OS does not understand the streaming dataflow or pipeline state in how they relate to the resource requirements of stream processing. Additionally, the lack of stream-awareness inhibits adaptive resource allocation in response to dynamic workload changes.<\/jats:p>\n                  <jats:p>We present Enjima, a modern SPE designed for scale-up on a single machine through adaptive stream-aware management of memory and compute resources. Enjima's eager, cache-aligned, block-based memory management avoids memory allocation on the critical path of system execution while providing efficient data transfer of events between streaming operators. Its variable batching forms event batches based on pending inputs and available output memory, reducing batching delays and memory accesses to enhance system performance. Enjima integrates a stream-aware, state-based operator scheduler that leverages fine-grained operator and pipeline metrics such as operator cost, selectivity, and latency gradient to optimize for both latency and throughput, enabling significant performance gains and rapid adaptation to dynamic workloads. Evaluation against state-of-the-art systems shows that Enjima achieves up to 6.3\u00d7 higher throughput and up to three orders of magnitude lower latency through integrated stream-aware memory and CPU resource management.<\/jats:p>","DOI":"10.1145\/3769790","type":"journal-article","created":{"date-parts":[[2025,12,6]],"date-time":"2025-12-06T04:32:13Z","timestamp":1764995533000},"page":"1-27","source":"Crossref","is-referenced-by-count":0,"title":["Enjima: A Resource-Adaptive Stream Processing System"],"prefix":"10.1145","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7169-233X","authenticated-orcid":false,"given":"Lasantha","family":"Fernando","sequence":"first","affiliation":[{"name":"Cheriton School of Computer Science, University of Waterloo, Waterloo, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-9152-5781","authenticated-orcid":false,"given":"Taebin","family":"Kim","sequence":"additional","affiliation":[{"name":"Cheriton School of Computer Science, University of Waterloo, Waterloo, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-7036-0397","authenticated-orcid":false,"given":"Khuzaima","family":"Daudjee","sequence":"additional","affiliation":[{"name":"Cheriton School of Computer Science, University of Waterloo, Waterloo, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-3335-8045","authenticated-orcid":false,"given":"Tilmann","family":"Rabl","sequence":"additional","affiliation":[{"name":"Hasso Plattner Institute, University of Potsdam, Potsdam, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,12,5]]},"reference":[{"key":"e_1_2_1_1_1","first-page":"277","volume-title":"CIDR","volume":"5","author":"Abadi Daniel J","year":"2005","unstructured":"Daniel J Abadi, Yanif Ahmad, Magdalena Balazinska, Ugur Cetintemel, Mitch Cherniack, Jeong-Hyon Hwang, Wolfgang Lindner, Anurag Maskey, Alex Rasin, Esther Ryvkina, et al., 2005. The design of the borealis stream processing engine.. In CIDR, Vol. 5. 277-289."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-003-0095-z"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.14778\/2824032.2824076"},{"key":"e_1_2_1_4_1","unstructured":"Apache Software Foundation. [n. d.] a. Apache Storm. https:\/\/storm.apache.org\/"},{"key":"e_1_2_1_5_1","unstructured":"Apache Software Foundation. [n. d.] b. Kafka Streams. https:\/\/kafka.apache.org\/39\/documentation\/streams\/"},{"key":"e_1_2_1_6_1","first-page":"480","volume-title":"Proceedings of the Thirtieth International Conference on Very Large Data Bases -","volume":"30","author":"Arasu Arvind","year":"2004","unstructured":"Arvind Arasu, Mitch Cherniack, Eduardo Galvez, David Maier, Anurag S. Maskey, Esther Ryvkina, Michael Stonebraker, and Richard Tibbetts. 2004. Linear road: a stream data management benchmark. In Proceedings of the Thirtieth International Conference on Very Large Data Bases - Volume 30 (Toronto, Canada) (VLDB '04). VLDB Endowment, 480-491."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3190664"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/872757.872789"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00138-024-01570-y"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11241-006-4961-9"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807128.1807148"},{"key":"e_1_2_1_12_1","first-page":"28","article-title":"Apache Flink: Stream and batch processing in a single engine","volume":"38","author":"Carbone Paris","year":"2015","unstructured":"Paris Carbone, Asterios Katsifodimos, Stephan Ewen, Volker Markl, Seif Haridi, and Kostas Tzoumas. 2015. Apache Flink: Stream and batch processing in a single engine. The Bulletin of the Technical Committee on Data Engineering, Vol. 38, 4 (2015), 28-38.","journal-title":"The Bulletin of the Technical Committee on Data Engineering"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3514496"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1016\/B978-012722442-8\/50079-3"},{"key":"e_1_2_1_15_1","unstructured":"Zongxiong Chen. 2019a. chenzongxiong\/Saber. https:\/\/github.com\/chenzongxiong\/Saber"},{"key":"e_1_2_1_16_1","unstructured":"Zongxiong Chen. 2019b. chenzongxiong\/streambox. GitHub. https:\/\/github.com\/chenzongxiong\/streambox"},{"key":"e_1_2_1_17_1","volume-title":"Boyang Jerry Peng, and Paul Poulosky","author":"Chintapalli Sanket","year":"2015","unstructured":"Sanket Chintapalli, Derek Dagit, Bobby Evans, Reza Farivar, Thomas Graves, Mark Holderbaugh, Zhuo Liu, Kyle Nusbaum, Kishorkumar Patil, Boyang Jerry Peng, and Paul Poulosky. 2015. Benchmarks for low latency (streaming) solutions including Apache Storm, Apache Spark, and Apache Flink. https:\/\/github.com\/yahoo\/streaming-benchmarks"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPSW.2016.138"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187671.2187677"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2670979.2670995"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1327452.1327492"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3514221.3517826"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3448016.3452794"},{"key":"e_1_2_1_24_1","unstructured":"Lasantha Fernando. 2025. Enjima codebase. https:\/\/github.com\/lasanthafdo\/enjima-lib"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-023-00819-8"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/776985.776986"},{"key":"e_1_2_1_27_1","unstructured":"Vincenzo Gulisano. 2020. Liebre stream processing engine. https:\/\/github.com\/vincenzo-gulisano\/Liebre"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/REAL.1991.160366"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675743.2772598"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2003.1260804"},{"key":"e_1_2_1_31_1","first-page":"103","volume-title":"Proceedings of 18th European Simulation Multiconference (ESM","author":"Karatza Helen D","year":"2004","unstructured":"Helen D Karatza. 2004. Epoch task scheduling in distributed server systems. In Proceedings of 18th European Simulation Multiconference (ESM 2004). EUROSIM, 103-108."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.14778\/2824032.2824132"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2882906"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1058150.1058158"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICESS.2009.14"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/321738.321743"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNSM.2014.2377295"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/223586.223610"},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the 2017 USENIX Conference on Usenix Annual Technical Conference (Santa Clara, CA, USA) (USENIX ATC '17). USENIX Association, USA, 617-629","author":"Miao Hongyu","year":"2017","unstructured":"Hongyu Miao, Heejin Park, Myeongjae Jeon, Gennady Pekhimenko, Kathryn S. McKinley, and Felix Xiaozhu Lin. 2017. StreamBox: modern stream processing on a multicore machine. In Proceedings of the 2017 USENIX Conference on Usenix Annual Technical Conference (Santa Clara, CA, USA) (USENIX ATC '17). USENIX Association, USA, 617-629."},{"key":"e_1_2_1_40_1","unstructured":"Aloysius Ka-Lau Mok. 1983. Fundamental design problems of distributed systems for the hard-real-time environment. Ph.D. Dissertation. Massachusetts Institute of Technology."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3328905.3329505"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3464298.3493407"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.5555\/1083592.1083756"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3062341.3062366"},{"key":"e_1_2_1_45_1","volume-title":"Proceedings of the ACS\/IEEE 2005 International Conference on Computer Systems and Applications (AICCSA '05)","author":"Sharaf M. A.","unstructured":"M. A. Sharaf, P. K. Chrysanthis, and A. Labrinidis. 2005. Preemptive rate-based operator scheduling in a data stream management system. In Proceedings of the ACS\/IEEE 2005 International Conference on Computer Systems and Applications (AICCSA '05). IEEE Computer Society, USA, 46-I."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.5555\/1182635.1164172"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/1331904.1331909"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/1107499.1107504"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3318464.3389753"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2020.2978480"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132750"},{"key":"e_1_2_1_52_1","unstructured":"Chris Whong. 2014. Foiling NYC's Taxi Trip Data. https:\/\/chriswhong.com\/open-data\/foil_nyc_taxi"},{"key":"e_1_2_1_53_1","first-page":"389","volume-title":"Move Fast and Meet Deadlines: Fine-grained Real-time Stream Processing with Cameo. In 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21)","author":"Xu Le","year":"2021","unstructured":"Le Xu, Shivaram Venkataraman, Indranil Gupta, Luo Mai, and Rahul Potharaju. 2021. Move Fast and Meet Deadlines: Fine-grained Real-time Stream Processing with Cameo. In 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21). USENIX Association, 389-405. https:\/\/www.usenix.org\/conference\/nsdi21\/presentation\/xu"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/2517349.2522737"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.14778\/3303753.3303758"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICAC.2016.27"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2017.119"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3300067"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3401025.3401758"}],"container-title":["Proceedings of the ACM on Management of Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3769790","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,13]],"date-time":"2026-06-13T04:45:02Z","timestamp":1781325902000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3769790"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,12,4]]},"references-count":59,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2025,12,4]]}},"alternative-id":["10.1145\/3769790"],"URL":"https:\/\/doi.org\/10.1145\/3769790","relation":{},"ISSN":["2836-6573"],"issn-type":[{"value":"2836-6573","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,12,4]]}}}