{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,3,1]],"date-time":"2025-03-01T06:01:06Z","timestamp":1740808866567,"version":"3.38.0"},"reference-count":37,"publisher":"Association for Computing Machinery (ACM)","issue":"2","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2024,10]]},"abstract":"<jats:p>Engineers rely heavily on observability tools to monitor their business and system metrics and set up alerting on it. A reliable and efficient monitoring system is very important for development velocity. In this paper, we introduce Goku, a time series database (TSDB) we built from the ground up at Pinterest. Over the years, we have studied user patterns and common requests to constantly evolve Goku to store and serve the use cases at Pinterest with high efficiency and reduced costs. At its core, Goku uses tiered storage to store new and frequently queried metrics data in memory while leveraging solid state drive (SSD) and hard disks (HDD) for older data. Goku aggregates metrics data at write time while also rolling up datapoints with lower time granularity to low latency to certain use cases. Goku also supports modifying configurations on metrics data like time to live (TTL), rollup granularity, backfilling capability, etc. Using multiple replicas and AWS S3 as backup, Goku is highly available and fault tolerant.<\/jats:p>","DOI":"10.14778\/3705829.3705862","type":"journal-article","created":{"date-parts":[[2025,2,28]],"date-time":"2025-02-28T23:21:06Z","timestamp":1740784866000},"page":"503-515","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Goku: A Schemaless Time Series Database for Large Scale Monitoring at Pinterest"],"prefix":"10.14778","volume":"18","author":[{"given":"Monil Mukesh","family":"Sanghavi","sequence":"first","affiliation":[{"name":"Pinterest Inc."}]},{"given":"Ming-May","family":"Hu","sequence":"additional","affiliation":[{"name":"Pinterest Inc."}]},{"given":"Zhenxiao","family":"Luo","sequence":"additional","affiliation":[{"name":"Pinterest Inc."}]},{"given":"Xiao","family":"Li","sequence":"additional","affiliation":[{"name":"Pinterest Inc."}]},{"given":"Kapil","family":"Bajaj","sequence":"additional","affiliation":[{"name":"Pinterest Inc."}]}],"member":"320","published-online":{"date-parts":[[2025,2,28]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Retrieved","year":"2024","unstructured":"[n.d.]. Apache HBase Book. Retrieved March 15, 2024 from https:\/\/hbase.apache.org\/book.html#arch.overview"},{"issue":"12","key":"e_1_2_1_2_1","first-page":"3181","article-title":"Monarch: Google's Planet-Scale In-Memory Time Series Database","volume":"13","author":"Adams Colin","year":"2020","unstructured":"Colin Adams, Luis Alonso, Benjamin Atkin, John Banning, Sumeer Bhola, Rick Buskens, Ming Chen, Xi Chen, Yoo Chung, Qin Jia, Nick Sakharov, George Talbot, Adam Tart, and Nick Taylor. 2020. Monarch: Google's Planet-Scale In-Memory Time Series Database. PVLDB, 13(12): 3181--3194 (2020). https:\/\/www.vldb.org\/pvldb\/vol13\/p3181-adams.pdf","journal-title":"PVLDB"},{"key":"e_1_2_1_3_1","unstructured":"M3 Authors. [n.d.]. Storage Engine. Retrieved March 14 2024 from https:\/\/m3db.io\/docs\/architecture\/m3db\/engine\/#m3tsz"},{"key":"e_1_2_1_4_1","volume-title":"Retrieved","author":"Authors Prometheus","year":"2024","unstructured":"Prometheus Authors. [n.d.]. Storage. THe Linux Foundation. Retrieved March 14, 2024 from https:\/\/prometheus.io\/docs\/prometheus\/1.8\/storage\/#chunk-encoding"},{"key":"e_1_2_1_5_1","volume-title":"Retrieved","author":"Authors DB","year":"2022","unstructured":"RocksDB Authors. 2022. A Tutorial of RocksDB SST formats. Retrieved March 14, 2024 from https:\/\/github.com\/facebook\/rocksdb\/wiki\/A-Tutorial-of-RocksDB-SST-formats"},{"key":"e_1_2_1_6_1","volume-title":"Retrieved","author":"Authors The","year":"2010","unstructured":"The OpenTSDB Authors. 2010. OpenTSDB Cardinality Explanation. Retrieved March 15, 2024 from http:\/\/opentsdb.net\/docs\/build\/html\/user_guide\/writing\/index.html#time-series-cardinality"},{"key":"e_1_2_1_7_1","volume-title":"Retrieved","author":"Authors The","year":"2010","unstructured":"The OpenTSDB Authors. 2010. OpenTSDB Overview. Retrieved March 15, 2024 from http:\/\/opentsdb.net\/overview.html"},{"key":"e_1_2_1_8_1","volume-title":"Retrieved","author":"Authors The","year":"2010","unstructured":"The OpenTSDB Authors. 2010. OpenTSDB Put Telnet. Retrieved March 15, 2024 from http:\/\/opentsdb.net\/docs\/build\/html\/api_telnet\/put.html"},{"key":"e_1_2_1_9_1","volume-title":"Retrieved","author":"Authors The","year":"2010","unstructured":"The OpenTSDB Authors. 2010. OpenTSDB Query. Retrieved March 15, 2024 from http:\/\/opentsdb.net\/docs\/build\/html\/user_guide\/query\/index.html"},{"key":"e_1_2_1_10_1","volume-title":"Retrieved","author":"Authors The","year":"2010","unstructured":"The OpenTSDB Authors. 2010. OpenTSDB Query Filters. Retrieved March 15, 2024 from http:\/\/opentsdb.net\/docs\/build\/html\/user_guide\/query\/filters.html"},{"key":"e_1_2_1_11_1","unstructured":"AWS. [n.d.]. AWS EFS. AWS. Retrieved March 14 2024 from https:\/\/aws.amazon.com\/efs\/"},{"key":"e_1_2_1_12_1","volume-title":"AWS. Retrieved","author":"AWS.","year":"2024","unstructured":"AWS. [n.d.]. What is Amazon S3? AWS. Retrieved March 14, 2024 from https:\/\/docs.aws.amazon.com\/AmazonS3\/latest\/userguide\/Welcome.html"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3318464.3386136"},{"key":"e_1_2_1_14_1","volume-title":"Confluent. Retrieved","year":"2024","unstructured":"Confluent. [n.d.]. Kafka Producer Batching. Confluent. Retrieved March 14, 2024 from https:\/\/docs.confluent.io\/kafka\/design\/producer-design.html#batching"},{"key":"e_1_2_1_15_1","volume-title":"std::basic_string_view. cppreference. Retrieved","year":"2024","unstructured":"cppreference. [n.d.]. std::basic_string_view. cppreference. Retrieved March 14, 2024 from https:\/\/en.cppreference.com\/w\/cpp\/string\/basic_string_view"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3483840"},{"key":"e_1_2_1_17_1","volume-title":"Retrieved","author":"Foundation Apache Software","year":"2023","unstructured":"Apache Software Foundation. 2023. Architecture. Retrieved March 15, 2024 from https:\/\/helix.apache.org\/Architecture.html"},{"key":"e_1_2_1_18_1","unstructured":"Patrick Hunt Mahadev Konar Flavio P. Junqueira and Benjamin Reed. [n.d.]. ZooKeeper: Wait-free coordination for Internet-scale systems. ([n. d.]). https:\/\/www.usenix.org\/legacy\/events\/atc10\/tech\/full_papers\/Hunt.pdf"},{"key":"e_1_2_1_19_1","volume-title":"InfluxData. Retrieved","year":"2024","unstructured":"InfluxData. [n.d.]. In-memory indexing and the Time-Structured Merge Tree (TSM). InfluxData. Retrieved March 14, 2024 from https:\/\/docs.influxdata.com\/influxdb\/v1\/concepts\/storage_engine\/#floats"},{"key":"e_1_2_1_20_1","unstructured":"InfluxData. [n.d.]. InfluxDB. Retrieved March 14 2024 from https:\/\/github.com\/influxdata\/influxdb"},{"key":"e_1_2_1_21_1","volume-title":"Kafka: a Distributed Messaging System for Log Processing. NetDB (June","author":"Kreps Jay","year":"2011","unstructured":"Jay Kreps, Neha Narkhede, and Jun Rao. 2011. Kafka: a Distributed Messaging System for Log Processing. NetDB (June 2011). https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2017\/09\/Kafka.pdf"},{"key":"e_1_2_1_22_1","volume-title":"Retrieved","author":"Learn Microsoft","year":"2024","unstructured":"Microsoft Learn. [n.d.]. Regular Expression Quick Language - Reference. Retrieved March 15, 2024 from https:\/\/learn.microsoft.com\/en-us\/dotnet\/standard\/base-types\/regular-expression-language-quick-reference"},{"key":"e_1_2_1_23_1","unstructured":"D. Lemire G. Ssi-Yan-Kai and O. Kaser. 2018. Consistently faster and smaller compressed bitmaps with Roaring. arXiv 4 (March 2018). https:\/\/arxiv.org\/pdf\/1603.06549.pdf"},{"key":"e_1_2_1_24_1","volume-title":"Thrift: Scalable Cross-Language Services Implementation. (April","author":"Mark Slee Aditya Agarwal","year":"2007","unstructured":"Aditya Agarwal Mark Slee and Marc Kwiatkowski. 2007. Thrift: Scalable Cross-Language Services Implementation. (April 2007). https:\/\/thrift.apache.org\/static\/files\/thrift-20070401.pdf"},{"key":"e_1_2_1_25_1","volume-title":"OpenTSDB. Retrieved","author":"TSDB.","year":"2024","unstructured":"OpenTSDB. [n.d.]. OpenTSDB http query format. OpenTSDB. Retrieved March 14, 2024 from http:\/\/opentsdb.net\/docs\/build\/html\/api_http\/query\/index.html#query-api-endpoints"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.14778\/2824032.2824078"},{"key":"e_1_2_1_27_1","unstructured":"pinterest. [n.d.]. Querybook. Retrieved March 14 2024 from https:\/\/www.querybook.org\/"},{"key":"e_1_2_1_28_1","volume-title":"Postgres. Retrieved","year":"2024","unstructured":"Postgres. [n.d.]. Postgres documentation. Postgres. Retrieved March 14, 2024 from https:\/\/www.postgresql.org\/docs\/"},{"key":"e_1_2_1_29_1","volume-title":"questdb. Retrieved","author":"DB.","year":"2024","unstructured":"questdb. [n.d.]. QuestDB. questdb. Retrieved March 14, 2024 from https:\/\/questdb.io\/docs\/"},{"key":"e_1_2_1_30_1","volume-title":"Retrieved","year":"2024","unstructured":"questdb. [n.d.]. QuestDB write batching. questdb. Retrieved March 14, 2024 from https:\/\/questdb.io\/docs\/reference\/sql\/insert\/"},{"key":"e_1_2_1_31_1","volume-title":"Retrieved","author":"DB.","year":"2024","unstructured":"RocksDB. 2024. Creating and Ingesting SST files. Retrieved March 14, 2024 from https:\/\/github.com\/facebook\/rocksdb\/wiki\/Creating-and-Ingesting-SST-files"},{"key":"e_1_2_1_32_1","unstructured":"superset. [n.d.]. Superset. Retrieved March 14 2024 from https:\/\/superset.apache.org\/"},{"key":"e_1_2_1_33_1","volume-title":"taosdata. Retrieved","year":"2024","unstructured":"taosdata. [n.d.]. TDEngine. taosdata. Retrieved March 14, 2024 from https:\/\/docs.tdengine.com\/"},{"key":"e_1_2_1_34_1","unstructured":"Timescale. [n.d.]. TimescaleDB. Timescale. Retrieved March 14 2024 from https:\/\/docs.timescale.com\/#TimescaleDB"},{"key":"e_1_2_1_35_1","volume-title":"Retrieved","year":"2024","unstructured":"Timescale. [n.d.]. TimescaleDB insert. timescale. Retrieved March 14, 2024 from https:\/\/docs.timescale.com\/use-timescale\/latest\/write-data\/insert\/#insert-multiple-rows"},{"key":"e_1_2_1_36_1","unstructured":"Timescale. [n.d.]. TSBS Benchmark. Retrieved March 14 2024 from https:\/\/github.com\/timescale\/tsbs"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.14778\/3415478.3415504"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3705829.3705862","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,2,28]],"date-time":"2025-02-28T23:22:24Z","timestamp":1740784944000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3705829.3705862"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10]]},"references-count":37,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,10]]}},"alternative-id":["10.14778\/3705829.3705862"],"URL":"https:\/\/doi.org\/10.14778\/3705829.3705862","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2024,10]]},"assertion":[{"value":"2025-02-28","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}