{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T17:34:19Z","timestamp":1783791259703,"version":"3.55.0"},"reference-count":50,"publisher":"Association for Computing Machinery (ACM)","issue":"6","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2021,2]]},"abstract":"<jats:p>Performance-monitoring timeseries systems such as Prometheus and InfluxDB play a critical role in assuring reliability and operationally. These systems commonly adopt a column-oriented storage model, by which timeseries samples from different time-series are separated, and all samples (with both numeric values and timestamps) in one timeseries are grouped into chunks and stored together. As a group of timeseries are often collected from the same source with the same timestamps, managing timestamps and metrics in a group manner provides more opportunities for query and insertion optimization but posts new challenges as well. Besides, for performance monitoring systems, to support better compression and efficient queries for most recent data that are most likely accessed by users, huge volumes of data are first cached in memory and then periodically flushed to disks. Periodic data flushing incurs high IO overhead, and simply discarding flushed data, which can still serve queries, not only is a waste but also brings huge memory reclamation cost. In this paper, we propose Heracles which integrates two techniques - (1) a new storage model, which enables efficient queries on compressed data by utilizing the shared timestamp column to easily locate corresponding metric values; (2) a novel two-level epoch-based memory manager, which allows the system to gradually flush and reclaim in-memory data while unreclaimed data can still serve queries. Heracles is implemented as a standalone module that can be easily integrated into existing performance monitoring timeseries systems. We have implemented a fully functional prototype with Heracles based on Prometheus tsdb, a representative open-source performance monitoring system, and conducted extensive experiments with real and synthetic timeseries data. Experimental results show that, compared with Prometheus, Heracles can improve the insertion throughput by 171%, and reduce the query latency and space usage by 32% and 30%, respectively, on average. Besides, to compare with other state-of-the-art storage techniques, we have integrated LevelDB (for LSM-tree-based structure) and Parquet (for column stores) into Prometheus tsdb, respectively, and experimental results show Heracles outperform these two integrations. We have released the open-source code of Heracles for public access.<\/jats:p>","DOI":"10.14778\/3447689.3447710","type":"journal-article","created":{"date-parts":[[2021,4,12]],"date-time":"2021-04-12T16:20:06Z","timestamp":1618244406000},"page":"1080-1092","source":"Crossref","is-referenced-by-count":14,"title":["Heracles"],"prefix":"10.14778","volume":"14","author":[{"given":"Zhiqi","family":"Wang","sequence":"first","affiliation":[{"name":"The Chinese University of Hong Kong"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jin","family":"Xue","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zili","family":"Shao","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,4,12]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1142473.1142548"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.14778\/3397230.3397236"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.14778\/2733085.2733096"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.14778\/2732951.2732958"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.5555\/2930583.2930587"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.5555\/1712666.1712668"},{"key":"e_1_2_1_7_1","volume-title":"USENIX Annual Technical Conference, FREENIX Track. 297--309","author":"Arcangeli Andrea","year":"2003"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/319996.319998"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3318464.3386136"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.14778\/2556549.2556575"},{"key":"e_1_2_1_11_1","unstructured":"Google Developers. 2020. Protocol Buffers - Base 128 Varints. https:\/\/developers.google.com\/protocol-buffers\/docs\/encoding#varints.  Google Developers. 2020. Protocol Buffers - Base 128 Varints. https:\/\/developers.google.com\/protocol-buffers\/docs\/encoding#varints."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2463676.2463710"},{"key":"e_1_2_1_13_1","unstructured":"Fackbook. 2020. Beringei - A high performance in memory time series storage engine. https:\/\/github.com\/facebookarchive\/beringei.  Fackbook. 2020. Beringei - A high performance in memory time series storage engine. https:\/\/github.com\/facebookarchive\/beringei."},{"key":"e_1_2_1_14_1","unstructured":"Apache Software Foundation. 2020. Apache Parquet. https:\/\/parquet.apache.org\/.  Apache Software Foundation. 2020. Apache Parquet. https:\/\/parquet.apache.org\/."},{"key":"e_1_2_1_15_1","unstructured":"The Apache Software Foundation. 2020. Apache HBase Project. https:\/\/hbase.apache.org\/.  The Apache Software Foundation. 2020. Apache HBase Project. https:\/\/hbase.apache.org\/."},{"key":"e_1_2_1_17_1","unstructured":"Sanjay Ghemawat and Jeff Dean. 2020. LevelDB. https:\/\/github.com\/google\/leveldb.  Sanjay Ghemawat and Jeff Dean. 2020. LevelDB. https:\/\/github.com\/google\/leveldb."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2008.167"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.23919\/ITC.2017.8064348"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpdc.2007.04.010"},{"key":"e_1_2_1_21_1","volume-title":"MonetDB: Two Decades of Research in Column-oriented Database Architectures","author":"Idreos Stratos"},{"key":"e_1_2_1_22_1","unstructured":"Docker Inc. 2020. Collect Docker metrics with Prometheus. https:\/\/docs.docker.com\/config\/thirdparty\/prometheus\/.  Docker Inc. 2020. Collect Docker metrics with Prometheus. https:\/\/docs.docker.com\/config\/thirdparty\/prometheus\/."},{"key":"e_1_2_1_23_1","unstructured":"InfluxData Inc. 2020. Flux data scripting language. https:\/\/docs.influxdata.com\/influxdb\/v2.0\/reference\/flux\/.  InfluxData Inc. 2020. Flux data scripting language. https:\/\/docs.influxdata.com\/influxdb\/v2.0\/reference\/flux\/."},{"key":"e_1_2_1_24_1","unstructured":"LogicMonitor Inc. 2020. LogicMonitor Case Studies. https:\/\/www.logicmonitor.com\/case-studies.  LogicMonitor Inc. 2020. LogicMonitor Case Studies. https:\/\/www.logicmonitor.com\/case-studies."},{"key":"e_1_2_1_25_1","unstructured":"influxdata. 2020. InfluxDB 1.7 Documentation. https:\/\/docs.influxdata.com\/influxdb\/.  influxdata. 2020. InfluxDB 1.7 Documentation. https:\/\/docs.influxdata.com\/influxdb\/."},{"key":"e_1_2_1_26_1","unstructured":"Influxdata. 2020. InfluxDB Query. https:\/\/docs.influxdata.com\/influxdb\/v2.0\/api\/#operation\/PatchDashboardsIDCellsIDView.  Influxdata. 2020. InfluxDB Query. https:\/\/docs.influxdata.com\/influxdb\/v2.0\/api\/#operation\/PatchDashboardsIDCellsIDView."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPCA.2011.6106531"},{"key":"e_1_2_1_28_1","unstructured":"William Kennedy. 2020. Scheduling In Go : Part II - Go Scheduler. https:\/\/www.ardanlabs.com\/blog\/2018\/08\/scheduling-in-go-part2.html.  William Kennedy. 2020. Scheduling In Go : Part II - Go Scheduler. https:\/\/www.ardanlabs.com\/blog\/2018\/08\/scheduling-in-go-part2.html."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.14778\/2367502.2367518"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2018.00026"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/571825.571829"},{"key":"e_1_2_1_32_1","unstructured":"OKLog. 2020. Universally Unique Lexicographically Sortable Identifier. https:\/\/github.com\/oklog\/ulid.  OKLog. 2020. Universally Unique Lexicographically Sortable Identifier. https:\/\/github.com\/oklog\/ulid."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.14778\/2824032.2824078"},{"key":"e_1_2_1_34_1","unstructured":"Bartlomiej Plotka. 2020. Thanos - Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project. https:\/\/github.com\/thanos-io\/thanos.  Bartlomiej Plotka. 2020. Thanos - Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project. https:\/\/github.com\/thanos-io\/thanos."},{"key":"e_1_2_1_35_1","unstructured":"Prometheus. 2020. Exporter for MySQL server metrics. https:\/\/github.com\/prometheus\/mysqld_exporter.  Prometheus. 2020. Exporter for MySQL server metrics. https:\/\/github.com\/prometheus\/mysqld_exporter."},{"key":"e_1_2_1_36_1","unstructured":"Prometheus. 2020. Node exporter - Exporter for machine metrics. https:\/\/github.com\/prometheus\/node_exporter.  Prometheus. 2020. Node exporter - Exporter for machine metrics. https:\/\/github.com\/prometheus\/node_exporter."},{"key":"e_1_2_1_37_1","unstructured":"Prometheus. 2020. Prometheus - Defining recording rules. https:\/\/prometheus.io\/docs\/prometheus\/latest\/configuration\/recording_rules\/.  Prometheus. 2020. Prometheus - Defining recording rules. https:\/\/prometheus.io\/docs\/prometheus\/latest\/configuration\/recording_rules\/."},{"key":"e_1_2_1_38_1","unstructured":"Prometheus. 2020. Prometheus - From metrics to insight power your metrics and alerting with a leading open-source monitoring solution. https:\/\/prometheus.io\/.  Prometheus. 2020. Prometheus - From metrics to insight power your metrics and alerting with a leading open-source monitoring solution. https:\/\/prometheus.io\/."},{"key":"e_1_2_1_39_1","unstructured":"Prometheus. 2020. Prometheus Range Queries. https:\/\/prometheus.io\/docs\/prometheus\/latest\/querying\/api\/#range-queries.  Prometheus. 2020. Prometheus Range Queries. https:\/\/prometheus.io\/docs\/prometheus\/latest\/querying\/api\/#range-queries."},{"key":"e_1_2_1_40_1","unstructured":"Prometheus. 2020. PromQL. https:\/\/prometheus.io\/docs\/prometheus\/latest\/querying\/basics\/.  Prometheus. 2020. PromQL. https:\/\/prometheus.io\/docs\/prometheus\/latest\/querying\/basics\/."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/PROC.1967.5493"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.5555\/898203"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.5555\/1268379.1268397"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.5555\/1083592.1083658"},{"key":"e_1_2_1_45_1","unstructured":"Yandex ClickHouse team. 2020. ClickHouse. https:\/\/clickhouse.tech\/.  Yandex ClickHouse team. 2020. ClickHouse. https:\/\/clickhouse.tech\/."},{"key":"e_1_2_1_46_1","unstructured":"Timescale. 2020. Time Series Benchmark Suite a tool for comparing and evaluating databases for time series data. https:\/\/github.com\/timescale\/tsbs.  Timescale. 2020. Time Series Benchmark Suite a tool for comparing and evaluating databases for time series data. https:\/\/github.com\/timescale\/tsbs."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/224964.224988"},{"key":"e_1_2_1_48_1","volume-title":"2020 USENIX Annual Technical Conference (USENIX ATC 20)","author":"Visheratin Alexander","year":"2020"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/502059.502057"},{"key":"e_1_2_1_50_1","unstructured":"Jason Wilder. 2020. simple8b Golang implementation. https:\/\/github.com\/jwilder\/encoding\/tree\/master\/simple8b.  Jason Wilder. 2020. simple8b Golang implementation. https:\/\/github.com\/jwilder\/encoding\/tree\/master\/simple8b."},{"key":"e_1_2_1_51_1","unstructured":"xitongsys. 2020. Pure golang library for reading\/writing parquet file. https:\/\/github.com\/xitongsys\/parquet-go.  xitongsys. 2020. Pure golang library for reading\/writing parquet file. https:\/\/github.com\/xitongsys\/parquet-go."}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3447689.3447710","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T11:21:46Z","timestamp":1672226506000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3447689.3447710"}},"subtitle":["an efficient storage model and data flushing for performance monitoring timeseries"],"short-title":[],"issued":{"date-parts":[[2021,2]]},"references-count":50,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2021,2]]}},"alternative-id":["10.14778\/3447689.3447710"],"URL":"https:\/\/doi.org\/10.14778\/3447689.3447710","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2021,2]]}}}