{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,2]],"date-time":"2026-08-02T11:14:28Z","timestamp":1785669268775,"version":"3.56.0"},"reference-count":72,"publisher":"Association for Computing Machinery (ACM)","issue":"11","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2023,7]]},"abstract":"<jats:p>Time series databases are essential for the large-scale deployment of many critical industrial applications. In infrastructure monitoring, for instance, a database system should be able to process large amounts of sensor data in real-time, execute continuous queries, and handle complex analytical queries such as anomaly detection or forecasting. Several benchmarks have been proposed to evaluate and understand how existing systems and design choices handle specific use cases and workloads. Unfortunately, none of them fully covers the peculiar requirements of monitoring applications. Furthermore, they fall short of providing an automated way to generate representative real-world data and workloads for testing and evaluating these systems.<\/jats:p>\n          <jats:p>We present TSM-Bench, a benchmark tailored for time series database systems used in monitoring applications. Our key contributions consist of (1) representative queries that meet the requirements that we collected from a water monitoring use case, and (2) a new scalable data generator method based on Generative Adversarial Networks (GAN) and Locality Sensitive Hashing (LSH). We demonstrate, through an extensive set of experiments, how TSM-Bench provides a comprehensive evaluation of the performance of seven leading time series database systems while offering a detailed characterization of their capabilities and trade-offs.<\/jats:p>","DOI":"10.14778\/3611479.3611532","type":"journal-article","created":{"date-parts":[[2023,8,25]],"date-time":"2023-08-25T02:08:08Z","timestamp":1692929288000},"page":"3363-3376","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":28,"title":["TSM-Bench: Benchmarking Time Series Database Systems for Monitoring Applications"],"prefix":"10.14778","volume":"16","author":[{"given":"Abdelouahab","family":"Khelifati","sequence":"first","affiliation":[{"name":"University of Fribourg, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mourad","family":"Khayati","sequence":"additional","affiliation":[{"name":"University of Fribourg, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anton","family":"Dign\u00f6s","sequence":"additional","affiliation":[{"name":"Free University of Bozen-Bolzano, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Djellel","family":"Difallah","sequence":"additional","affiliation":[{"name":"NYU Abu Dhabi, United Arab Emirates"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Philippe","family":"Cudr\u00e9-Mauroux","sequence":"additional","affiliation":[{"name":"University of Fribourg, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,8,24]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/FMEC.2019.8795363"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1111\/jtsa.12000"},{"key":"e_1_2_1_3_1","unstructured":"Alexandr Andoni Piotr Indyk Thijs Laarhoven Ilya P. Razenshteyn and Ludwig Schmidt. 2015. Practical and Optimal LSH for Angular Distance. (2015) 1225--1233. https:\/\/proceedings.neurips.cc\/paper\/2015\/hash\/2823f4797102ce1a1aec05359cc16dd9-Abstract.html  Alexandr Andoni Piotr Indyk Thijs Laarhoven Ilya P. Razenshteyn and Ludwig Schmidt. 2015. Practical and Optimal LSH for Angular Distance. (2015) 1225--1233. https:\/\/proceedings.neurips.cc\/paper\/2015\/hash\/2823f4797102ce1a1aec05359cc16dd9-Abstract.html"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611973402.76"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2668930.2688055"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijforecast.2015.07.002"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2014.2308321"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.14778\/3565838.3565840"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.14778\/3523210.3523222"},{"key":"e_1_2_1_10_1","unstructured":"ClickHouse. 2022. ClickBench. https:\/\/github.com\/ClickHouse\/ClickBench\/.  ClickHouse. 2022. ClickBench. https:\/\/github.com\/ClickHouse\/ClickBench\/."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807128.1807152"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/0895-4356(90)90005-A"},{"key":"e_1_2_1_13_1","volume-title":"Water relations of drought-stressed temperate trees benefit from short drought-intermitting rainfall events. Agricultural and forest meteorology 265","author":"Dietrich Lars","year":"2019","unstructured":"Lars Dietrich and Ansgar Kahmen . 2019. Water relations of drought-stressed temperate trees benefit from short drought-intermitting rainfall events. Agricultural and forest meteorology 265 ( 2019 ), 70--77. Lars Dietrich and Ansgar Kahmen. 2019. Water relations of drought-stressed temperate trees benefit from short drought-intermitting rainfall events. Agricultural and forest meteorology 265 (2019), 70--77."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.14778\/2732240.2732246"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.14778\/1454159.1454226"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2212.13310"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.14778\/3476311.3476407"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.14778\/3229863.3229878"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.14778\/3407790.3407802"},{"key":"e_1_2_1_20_1","volume-title":"Statistical time series methods for structural health monitoring. Encyclopedia of structural health monitoring","author":"Fassois Spilios D","year":"2009","unstructured":"Spilios D Fassois and John S Sakellariou . 2009. Statistical time series methods for structural health monitoring. Encyclopedia of structural health monitoring ( 2009 ), 443--472. Spilios D Fassois and John S Sakellariou. 2009. Statistical time series methods for structural health monitoring. Encyclopedia of structural health monitoring (2009), 443--472."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2017.106"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.3390\/rs13112196"},{"key":"e_1_2_1_23_1","volume-title":"RobustTAD: Robust Time Series Anomaly Detection via Decomposition and Convolutional Neural Networks. CoRR abs\/2002.09545","author":"Gao Jingkun","year":"2020","unstructured":"Jingkun Gao , Xiaomin Song , Qingsong Wen , Pichao Wang , Liang Sun , and Huan Xu. 2020. RobustTAD: Robust Time Series Anomaly Detection via Decomposition and Convolutional Neural Networks. CoRR abs\/2002.09545 ( 2020 ). arXiv:2002.09545 https:\/\/arxiv.org\/abs\/2002.09545 Jingkun Gao, Xiaomin Song, Qingsong Wen, Pichao Wang, Liang Sun, and Huan Xu. 2020. RobustTAD: Robust Time Series Anomaly Detection via Decomposition and Convolutional Neural Networks. CoRR abs\/2002.09545 (2020). arXiv:2002.09545 https:\/\/arxiv.org\/abs\/2002.09545"},{"key":"e_1_2_1_24_1","volume-title":"Benchmark Handbook: For Database and Transaction Processing Systems","author":"Gray Jim","year":"1992","unstructured":"Jim Gray . 1992 . Benchmark Handbook: For Database and Transaction Processing Systems . Morgan Kaufmann Publishers Inc . Jim Gray. 1992. Benchmark Handbook: For Database and Transaction Processing Systems. Morgan Kaufmann Publishers Inc."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.14778\/3407790.3407791"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.14778\/3421424.3421429"},{"key":"e_1_2_1_27_1","unstructured":"InfluxData. 2016. InfluxDB comparison. https:\/\/github.com\/influxdata\/influxdb-comparisons.  InfluxData. 2016. InfluxDB comparison. https:\/\/github.com\/influxdata\/influxdb-comparisons."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3538712.3538723"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2017.2740932"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2021.3103648"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10452-016-9576-1"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/GIOTS49054.2020.9119497"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1002\/sam.11461"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3221269.3221293"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.5555\/3430915.3442429"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-019-01421-7"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.14778\/3377369.3377383"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/BigData47090.2019.9005580"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE51399.2021.00298"},{"key":"e_1_2_1_40_1","volume-title":"Djellel Eddine Difallah, and Philippe Cudr\u00e9-Mauroux","author":"Khelifati Abdelouahab","year":"2023","unstructured":"Abdelouahab Khelifati , Mourad Khayati , Anton Dign\u00f6s , Djellel Eddine Difallah, and Philippe Cudr\u00e9-Mauroux . 2023 . TSM-Bench: Benchmarking Time Series Database Systems for Monitoring Applications [Technical Report ]. https:\/\/github.com\/eXascaleInfolab\/TSM-Bench Abdelouahab Khelifati, Mourad Khayati, Anton Dign\u00f6s, Djellel Eddine Difallah, and Philippe Cudr\u00e9-Mauroux. 2023. TSM-Bench: Benchmarking Time Series Database Systems for Monitoring Applications [Technical Report]. https:\/\/github.com\/eXascaleInfolab\/TSM-Bench"},{"key":"e_1_2_1_41_1","volume-title":"Thirty years of river restoration in Switzerland: implemented measures and lessons learned. Environmental earth sciences 72, 6","author":"Kurth Anne-Marie","year":"2014","unstructured":"Anne-Marie Kurth and Mario Schirmer . 2014. Thirty years of river restoration in Switzerland: implemented measures and lessons learned. Environmental earth sciences 72, 6 ( 2014 ), 2065--2079. Anne-Marie Kurth and Mario Schirmer. 2014. Thirty years of river restoration in Switzerland: implemented measures and lessons learned. Environmental earth sciences 72, 6 (2014), 2065--2079."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2015.2457240"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.14778\/2556549.2556552"},{"key":"e_1_2_1_44_1","unstructured":"Rui Liu and Jun Yuan. 2019. Benchmarking Time Series Databases with IoTDB-Benchmark for IoT Scenarios. (2019). arXiv:1901.08304 [cs.DB]  Rui Liu and Jun Yuan. 2019. Benchmarking Time Series Databases with IoTDB-Benchmark for IoT Scenarios. (2019). arXiv:1901.08304 [cs.DB]"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-70139-4"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSTARS.2016.2517118"},{"key":"e_1_2_1_47_1","volume-title":"The fastest similarity search algorithm for time series subsequences under Euclidean distance. url: www cs unm edu\/ mueen. FastestSimilaritySearch html (Accessed","author":"Mueen Abdullah","year":"2016","unstructured":"Abdullah Mueen , Krishnamurthy Viswanathan , CK Gupta , and Eamonn Keogh . 2015. The fastest similarity search algorithm for time series subsequences under Euclidean distance. url: www cs unm edu\/ mueen. FastestSimilaritySearch html (Accessed 24 May 2016 ) (2015). Abdullah Mueen, Krishnamurthy Viswanathan, CK Gupta, and Eamonn Keogh. 2015. The fastest similarity search algorithm for time series subsequences under Euclidean distance. url: www cs unm edu\/ mueen. FastestSimilaritySearch html (Accessed 24 May 2016) (2015)."},{"key":"e_1_2_1_48_1","volume-title":"Universit\u00e9 Libre de Bruxelles 12","author":"Zehra Naqvi Syeda Noor","year":"2017","unstructured":"Syeda Noor Zehra Naqvi , Sofia Yfantidou , and Esteban Zim\u00e1nyi . 2017. Time series databases and influxdb. Studienarbeit , Universit\u00e9 Libre de Bruxelles 12 ( 2017 ). Syeda Noor Zehra Naqvi, Sofia Yfantidou, and Esteban Zim\u00e1nyi. 2017. Time series databases and influxdb. Studienarbeit, Universit\u00e9 Libre de Bruxelles 12 (2017)."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.14778\/3021924.3021933"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00027-019-0626-3"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.14778\/3342263.3342648"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.14778\/3529337.3529354"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.14778\/3231751.3231757"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.14778\/3570690.3570694"},{"key":"e_1_2_1_55_1","unstructured":"Alec Radford Luke Metz and Soumith Chintala. 2016. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. (2016). http:\/\/arxiv.org\/abs\/1511.06434  Alec Radford Luke Metz and Soumith Chintala. 2016. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. (2016). http:\/\/arxiv.org\/abs\/1511.06434"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSNIP.2014.6827606"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/I2MTC.2019.8827164"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.14778\/3236187.3236214"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.14778\/3538598.3538602"},{"key":"e_1_2_1_60_1","unstructured":"Timescale. 2020. Time Series Benchmark Suite. https:\/\/github.com\/timescale\/tsbs  Timescale. 2020. Time Series Benchmark Suite. https:\/\/github.com\/timescale\/tsbs"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3439723"},{"key":"e_1_2_1_62_1","volume-title":"Time Series Data Augmentation for Deep Learning: A Survey. CoRR abs\/2002.12478","author":"Wen Qingsong","year":"2020","unstructured":"Qingsong Wen , Liang Sun , Xiaomin Song , Jingkun Gao , Xue Wang , and Huan Xu. 2020. Time Series Data Augmentation for Deep Learning: A Survey. CoRR abs\/2002.12478 ( 2020 ). arXiv:2002.12478 https:\/\/arxiv.org\/abs\/2002.12478 Qingsong Wen, Liang Sun, Xiaomin Song, Jingkun Gao, Xue Wang, and Huan Xu. 2020. Time Series Data Augmentation for Deep Learning: A Survey. CoRR abs\/2002.12478 (2020). arXiv:2002.12478 https:\/\/arxiv.org\/abs\/2002.12478"},{"key":"e_1_2_1_63_1","volume-title":"Time Series Anomaly Detection Using Convolutional Neural Networks and Transfer Learning. CoRR abs\/1905.13628","author":"Wen Tailai","year":"2019","unstructured":"Tailai Wen and Roy Keyes . 2019. Time Series Anomaly Detection Using Convolutional Neural Networks and Transfer Learning. CoRR abs\/1905.13628 ( 2019 ). arXiv:1905.13628 http:\/\/arxiv.org\/abs\/1905.13628 Tailai Wen and Roy Keyes. 2019. Time Series Anomaly Detection Using Convolutional Neural Networks and Transfer Learning. CoRR abs\/1905.13628 (2019). arXiv:1905.13628 http:\/\/arxiv.org\/abs\/1905.13628"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.14778\/3503585.3503604"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.14778\/3547305.3547319"},{"key":"e_1_2_1_66_1","unstructured":"Yahoo. 2023. Yahoo Cloud Server Benchmark for Time Series. https:\/\/github.com\/TSDBBench\/YCSB-TS.  Yahoo. 2023. Yahoo Cloud Server Benchmark for Time Series. https:\/\/github.com\/TSDBBench\/YCSB-TS."},{"key":"e_1_2_1_67_1","unstructured":"Byoung-Kee Yi and Christos Faloutsos. 2000. Fast Time Sequence Indexing for Arbitrary Lp Norms. (2000) 385--394. http:\/\/www.vldb.org\/conf\/2000\/P385.pdf  Byoung-Kee Yi and Christos Faloutsos. 2000. Fast Time Sequence Indexing for Arbitrary Lp Norms. (2000) 385--394. http:\/\/www.vldb.org\/conf\/2000\/P385.pdf"},{"key":"e_1_2_1_68_1","volume-title":"TS-Benchmark: A Benchmark for Time Series Databases. 2021 IEEE 37th International Conference on Data Engineering (ICDE)","author":"Yuanzhe Hao","year":"2021","unstructured":"Hao Yuanzhe , Qin Xiongpai , Chen Yueguo , Li Yaru , Sun Xiaoguang , Tao Yu , Zhang Xiao , and Du Xiaoyong . 2021 . TS-Benchmark: A Benchmark for Time Series Databases. 2021 IEEE 37th International Conference on Data Engineering (ICDE) (2021). Hao Yuanzhe, Qin Xiongpai, Chen Yueguo, Li Yaru, Sun Xiaoguang, Tao Yu, Zhang Xiao, and Du Xiaoyong. 2021. TS-Benchmark: A Benchmark for Time Series Databases. 2021 IEEE 37th International Conference on Data Engineering (ICDE) (2021)."},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0925-2312(01)00702-0"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1109\/IWQOS52092.2021.9521351"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1088\/1742-6596\/1944\/1\/012012"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.14778\/2824032.2824099"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3611479.3611532","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,23]],"date-time":"2023-09-23T22:14:27Z","timestamp":1695507267000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3611479.3611532"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7]]},"references-count":72,"journal-issue":{"issue":"11","published-print":{"date-parts":[[2023,7]]}},"alternative-id":["10.14778\/3611479.3611532"],"URL":"https:\/\/doi.org\/10.14778\/3611479.3611532","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2023,7]]},"assertion":[{"value":"2023-08-24","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}