{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,5,10]],"date-time":"2025-05-10T23:01:16Z","timestamp":1746918076311},"reference-count":53,"publisher":"Association for Computing Machinery (ACM)","issue":"12","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2021,7]]},"abstract":"<jats:p>Some mission critical systems, e.g., fraud detection, require accurate, real-time metrics over long time sliding windows on applications that demand high throughput and low latencies. As these applications need to run \"forever\" and cope with large, spiky data loads, they further require to be run in a distributed setting. We are unaware of any streaming system that provides all those properties. Instead, existing systems take large simplifications, such as implementing sliding windows as a fixed set of overlapping windows, jeopardizing metric accuracy (violating regulatory rules) or latency (breaching service agreements). In this paper, we propose Railgun, a fault-tolerant, elastic, and distributed streaming system supporting real-time sliding windows for scenarios requiring high loads and millisecond-level latencies. We benchmarked an initial prototype of Railgun using real data, showing significant lower latency than Flink and low memory usage independent of window size. Further, we show that Railgun scales nearly linearly, respecting our msec-level latencies at high percentiles (&lt;250ms @ 99.9%) even under a load of 1 million events per second.<\/jats:p>","DOI":"10.14778\/3476311.3476384","type":"journal-article","created":{"date-parts":[[2021,10,28]],"date-time":"2021-10-28T22:48:43Z","timestamp":1635461323000},"page":"3069-3082","source":"Crossref","is-referenced-by-count":1,"title":["Railgun"],"prefix":"10.14778","volume":"14","author":[{"given":"Ana Sofia","family":"Gomes","sequence":"first","affiliation":[{"name":"Feedzai"}]},{"given":"Jo\u00e3o","family":"Oliveirinha","sequence":"additional","affiliation":[{"name":"Feedzai"}]},{"given":"Pedro","family":"Cardoso","sequence":"additional","affiliation":[{"name":"Feedzai"}]},{"given":"Pedro","family":"Bizarro","sequence":"additional","affiliation":[{"name":"Feedzai"}]}],"member":"320","published-online":{"date-parts":[[2021,10,28]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"The Design of the Borealis Stream Processing Engine. In CIDR","author":"Abadi D. J.","year":"2005"},{"key":"e_1_2_1_2_1","volume-title":"Zdonik","author":"Abadi Daniel J.","year":"2005"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/872757.872855"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.14778\/2824032.2824076"},{"key":"e_1_2_1_5_1","unstructured":"Alibaba. 2020. Alibaba Cloud Supported 583 000 Orders\/Second for 2020 Double 11. https:\/\/www.alibabacloud.com\/blog\/alibaba-cloud-supported-583000-orderssecond-for-2020-double-11-the-highest-traffic-peak-in-the-world_596884.  Alibaba. 2020. Alibaba Cloud Supported 583 000 Orders\/Second for 2020 Double 11. https:\/\/www.alibabacloud.com\/blog\/alibaba-cloud-supported-583000-orderssecond-for-2020-double-11-the-highest-traffic-peak-in-the-world_596884."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/872757.872854"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3190664"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/543613.543615"},{"key":"e_1_2_1_9_1","volume-title":"Disco: Efficient Distributed Window Aggregation. In EDBT'20","author":"Benson L."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3394486"},{"key":"e_1_2_1_11_1","first-page":"28","article-title":"Apache Flink\u2122: Stream and Batch Processing in a Single Engine","volume":"38","author":"Carbone Paris","year":"2015","journal-title":"IEEE Data Eng. Bull."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2983323.2983807"},{"key":"e_1_2_1_13_1","first-page":"51","article-title":"Trill: Engineering a Library for Diverse Analytics","volume":"38","author":"Chandramouli Badrish","year":"2015","journal-title":"IEEE Data Eng. Bull."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/872757.872857"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/CCGRID.2018.00029"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/342009.335432"},{"key":"e_1_2_1_17_1","unstructured":"Apache Commons. 2011. JEXEL Expressions. https:\/\/commons.apache.org\/proper\/commons-jexl\/.  Apache Commons. 2011. JEXEL Expressions. https:\/\/commons.apache.org\/proper\/commons-jexl\/."},{"key":"e_1_2_1_18_1","unstructured":"Confluent. 2016. Kafka Streams. https:\/\/www.confluent.io\/blog\/introducing-kafka-streams-stream-processing-made-simple\/.  Confluent. 2016. Kafka Streams. https:\/\/www.confluent.io\/blog\/introducing-kafka-streams-stream-processing-made-simple\/."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2408776.2408794"},{"key":"e_1_2_1_20_1","unstructured":"Facebook. 2012. RocksDB. https:\/\/rocksdb.org\/.  Facebook. 2012. RocksDB. https:\/\/rocksdb.org\/."},{"key":"e_1_2_1_21_1","volume-title":"Custom Window Processing. https:\/\/flink.apache.org\/news\/2020\/07\/30\/demo-fraud-detection-3.html.","author":"Fedulov A."},{"key":"e_1_2_1_22_1","volume-title":"Fourth Biennial Conference on Innovative Data Systems Research, CIDR 2009, Asilomar, CA, USA, January 4--7, 2009, Online Proceedings. www.cidrdb.org. http:\/\/www-db.cs.wisc.edu\/cidr\/cidr2009\/Paper_122","author":"Franklin Michael J.","year":"2009"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1376616.1376729"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/776985.776986"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2528412"},{"key":"e_1_2_1_26_1","unstructured":"T. Hoff. 2015. Your Load Generator Is Probably Lying To You - Take The Red Pill And Find Out Why. http:\/\/highscalability.com\/blog\/2015\/10\/5\/your-load-generator-is-probably-lying-to-you-take-the-red-pi.html.  T. Hoff. 2015. Your Load Generator Is Probably Lying To You - Take The Red Pill And Find Out Why. http:\/\/highscalability.com\/blog\/2015\/10\/5\/your-load-generator-is-probably-lying-to-you-take-the-red-pi.html."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2018.00169"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4684-2001-2_9"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/BigData.2015.7364082"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.5555\/260999"},{"key":"e_1_2_1_31_1","first-page":"1","article-title":"Kafka: A distributed messaging system for log processing","volume":"11","author":"Kreps Jay","year":"2011","journal-title":"Proceedings of the NetDB"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2742788"},{"key":"e_1_2_1_33_1","unstructured":"Lightbend. 2011. Akka Streams. https:\/\/doc.akka.io\/docs\/akka\/current\/stream\/index.html.  Lightbend. 2011. Akka Streams. https:\/\/doc.akka.io\/docs\/akka\/current\/stream\/index.html."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2479871.2479933"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-22970-1_3"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.14778\/3137765.3137770"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.14778\/3137765.3137770"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1007\/s002360050048"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3328905.3338224"},{"key":"e_1_2_1_40_1","unstructured":"John Rose. 2012. CompressedOops. https:\/\/wiki.openjdk.java.net\/display\/HotSpot\/CompressedOops.  John Rose. 2012. CompressedOops. https:\/\/wiki.openjdk.java.net\/display\/HotSpot\/CompressedOops."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipl.2006.11.009"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3310361"},{"key":"e_1_2_1_43_1","unstructured":"Pivotal Software. 2007. RabbitMQ. https:\/\/www.rabbitmq.com\/.  Pivotal Software. 2007. RabbitMQ. https:\/\/www.rabbitmq.com\/."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2110486.2110493"},{"key":"e_1_2_1_45_1","unstructured":"Sybase. 2009. Coral8 Integration Guide. http:\/\/infocenter-archive.sybase.com\/help\/topic\/com.sybase.infocenter.dc01030.0200\/pdf\/cep-IntegrationGuide.pdf.  Sybase. 2009. Coral8 Integration Guide. http:\/\/infocenter-archive.sybase.com\/help\/topic\/com.sybase.infocenter.dc01030.0200\/pdf\/cep-IntegrationGuide.pdf."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3318464.3389753"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2595641"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2018.00135"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2003.1198390"},{"key":"e_1_2_1_50_1","doi-asserted-by":"crossref","unstructured":"Author(s) B. P. Welford and B. P. Welford. 1962. Note on a method for calculating corrected sums of squares and products. Technometrics (1962) 419--420.  Author(s) B. P. Welford and B. P. Welford. 1962. Note on a method for calculating corrected sums of squares and products. Technometrics (1962) 419--420.","DOI":"10.1080\/00401706.1962.10490022"},{"key":"e_1_2_1_51_1","volume-title":"Project Tungsten: Bringing Apache Spark Closer to Bare Metal. https:\/\/databricks.com\/blog\/2015\/04\/28\/project-tungsten-bringing-spark-closer-to-bare-metal.html.","author":"Xin Reynold","year":"2015"},{"key":"e_1_2_1_52_1","unstructured":"Andrey Zagrebin. 21 April 2020. Memory Management Improvements with Apache Flink 1.10. https:\/\/flink.apache.org\/news\/2020\/04\/21\/memory-management-improvements-flink-1.10.html.  Andrey Zagrebin. 21 April 2020. Memory Management Improvements with Apache Flink 1.10. https:\/\/flink.apache.org\/news\/2020\/04\/21\/memory-management-improvements-flink-1.10.html."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2017.166"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3476311.3476384","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T11:38:40Z","timestamp":1672227520000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3476311.3476384"}},"subtitle":["managing large streaming windows under MAD requirements"],"short-title":[],"issued":{"date-parts":[[2021,7]]},"references-count":53,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2021,7]]}},"alternative-id":["10.14778\/3476311.3476384"],"URL":"https:\/\/doi.org\/10.14778\/3476311.3476384","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2021,7]]}}}