{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,6]],"date-time":"2026-08-06T18:17:01Z","timestamp":1786040221514,"version":"3.56.0"},"reference-count":43,"publisher":"Association for Computing Machinery (ACM)","issue":"12","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2024,8]]},"abstract":"<jats:p>The proliferation of the Internet of Things (IoT) has led to an exponential increase in time series data, distributed and applied in various contexts, demanding a dedicated storage solution. Based on our observations and analysis of IoT production systems, we have characterized 3 requirements for time series data: (1) a close association with devices and sensors, (2) continually synchronizing between cloud-edge, and (3) requiring the ability for high ingestion and low latency access on big volume data. Despite the growing trend, current time series database systems lack a standardized file format, and existing open file formats do not adequately leverage the unique characteristics of IoT time series data. In this paper, we introduce Apache TsFile, a specialized file format tailored for IoT time series data. TsFile organizes data by devices, creating indexes based on device-related information. Our experiments demonstrate the efficiency of TsFile in achieving high data ingestion rates, minimizing latency, and optimizing data compactness.<\/jats:p>","DOI":"10.14778\/3685800.3685827","type":"journal-article","created":{"date-parts":[[2024,11,8]],"date-time":"2024-11-08T17:25:21Z","timestamp":1731086721000},"page":"4064-4076","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Apache TsFile: An IoT-Native Time Series File Format"],"prefix":"10.14778","volume":"17","author":[{"given":"Xin","family":"Zhao","sequence":"first","affiliation":[{"name":"Tsinghua University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jialin","family":"Qiao","sequence":"additional","affiliation":[{"name":"Timecho Ltd"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiangdong","family":"Huang","sequence":"additional","affiliation":[{"name":"Tsinghua University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chen","family":"Wang","sequence":"additional","affiliation":[{"name":"Timecho Ltd"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shaoxu","family":"Song","sequence":"additional","affiliation":[{"name":"Tsinghua University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jianmin","family":"Wang","sequence":"additional","affiliation":[{"name":"Tsinghua University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,11,8]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Apache. 2024. https:\/\/github.com\/apache\/parquet-java\/."},{"key":"e_1_2_1_2_1","unstructured":"Apache. 2024. https:\/\/arrow.apache.org\/docs\/cpp\/gandiva.html."},{"key":"e_1_2_1_3_1","unstructured":"Apache. 2024. https:\/\/arrow.apache.org\/docs\/java\/dataset.html."},{"key":"e_1_2_1_4_1","unstructured":"Apache Thrift. 2024. https:\/\/thrift.apache.org\/."},{"key":"e_1_2_1_5_1","unstructured":"Aveva. 2024. Asset Framework and PI System Explorer. https:\/\/docs.aveva.com\/bundle\/pi-server-af-pse-f\/page\/1031642.html. Accessed: 2024-02-16."},{"key":"e_1_2_1_6_1","doi-asserted-by":"crossref","unstructured":"DC Barr. 1994. The use of a data historian to extend plant life. (1994).","DOI":"10.1049\/cp:19941091"},{"key":"e_1_2_1_7_1","volume-title":"Carsten Magerkurth, Andreas Nettstr\u00e4ter, Julinda Stefa, and Joachim W Walewski.","author":"Bauer Martin","year":"2013","unstructured":"Martin Bauer, Nicola Bui, Jourik De Loof, Carsten Magerkurth, Andreas Nettstr\u00e4ter, Julinda Stefa, and Joachim W Walewski. 2013. IoT reference model. Enabling Things to Talk: Designing IoT solutions with the IoT architectural reference model (2013), 113--162."},{"key":"e_1_2_1_8_1","unstructured":"benchANT. 2024. https:\/\/benchant.com\/ranking\/database-ranking."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1629175.1629198"},{"key":"e_1_2_1_10_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3588703","article-title":"Grouping time series for efficient columnar storage","volume":"1","author":"Fang Chenguang","year":"2023","unstructured":"Chenguang Fang, Shaoxu Song, Haoquan Guan, Xiangdong Huang, Chen Wang, and Jianmin Wang. 2023. Grouping time series for efficient columnar storage. Proceedings of the ACM on Management of Data 1, 1 (2023), 1--26.","journal-title":"Proceedings of the ACM on Management of Data"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.14778\/3538598.3538607"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3534678.3539373"},{"key":"e_1_2_1_13_1","unstructured":"GE Digital. 2022. About Asset Model. https:\/\/www.ge.com\/digital\/documentation\/predix-essentials\/latest\/c_apm_asset_about_asset_model_1.html. Accessed: 2024-02-16."},{"key":"e_1_2_1_14_1","unstructured":"The PostgreSQL Global Development Group. 2024. https:\/\/www.postgresql.org\/docs\/16\/storage-file-layout.html\/."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2011.5767933"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2595630"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.14778\/2556549.2556559"},{"key":"e_1_2_1_18_1","unstructured":"InfluxData. 2024. https:\/\/www.influxdata.com\/time-series-platform\/influxdb\/."},{"key":"e_1_2_1_19_1","unstructured":"InfluxData. 2024. https:\/\/www.influxdata.com\/blog\/understanding-influxdb-iox-commitment-open-source\/."},{"key":"e_1_2_1_20_1","unstructured":"InfluxData. 2024. https:\/\/github.com\/influxdata\/influxdb\/tree\/3c5e5bf241dcc2c0e13554c5286577ad6066bfec\/parquet_file\/."},{"key":"e_1_2_1_21_1","volume-title":"Batch Control Part 1: Models and Terminology","author":"International Society of Automation (ISA) 2010.","year":"2010","unstructured":"International Society of Automation (ISA) 2010. Batch Control Part 1: Models and Terminology. International Society of Automation (ISA). Accessed online at https:\/\/www.isa.org\/products\/ansi-isa-88-00-01-2010-batch-control-part-1-models."},{"key":"e_1_2_1_22_1","volume-title":"Enterprise-Control System Integration","author":"International Society of Automation (ISA) 2010.","year":"2010","unstructured":"International Society of Automation (ISA) 2010. Enterprise-Control System Integration. International Society of Automation (ISA). Accessed online at https:\/\/www.isa.org\/products\/ansi-isa-95-00-01-2010-iec-62264-1-mod-enterprise."},{"key":"e_1_2_1_23_1","volume-title":"Workshop on Data Mining Applications in Sustainability (SIGKDD)","volume":"25","author":"Zico Kolter J","year":"2011","unstructured":"J Zico Kolter and Matthew J Johnson. 2011. REDD: A public data set for energy disaggregation research. In Workshop on Data Mining Applications in Sustainability (SIGKDD), San Diego, CA, Vol. 25. 59--62."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.14778\/3611479.3611507"},{"key":"e_1_2_1_25_1","unstructured":"Wes McKinney. 2024. https:\/\/github.com\/wesm\/feather\/."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.14778\/1920841.1920886"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/S002360050048"},{"key":"e_1_2_1_28_1","unstructured":"Apache Parquet. 2024. https:\/\/parquet.apache.org\/."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.14778\/2824032.2824078"},{"key":"e_1_2_1_30_1","unstructured":"QuestDB. 2024. https:\/\/questdb.io\/docs\/concept\/storage-model\/."},{"key":"e_1_2_1_31_1","unstructured":"SQLite. 2024. https:\/\/www.sqlite.org\/."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE48307.2020.00068"},{"key":"e_1_2_1_33_1","unstructured":"TimescaleDB. 2023. https:\/\/www.timescale.com\/."},{"key":"e_1_2_1_34_1","unstructured":"TimescaleDB. 2023. Time Series Benchmark Suite (TSBS). https:\/\/github.com\/timescale\/tsbs."},{"key":"e_1_2_1_35_1","unstructured":"VictoriaMetrics. 2024. https:\/\/docs.victoriametrics.com\/#storage\/."},{"key":"e_1_2_1_36_1","first-page":"1","article-title":"Apache IoTDB: A Time Series Database for IoT Applications","volume":"1","author":"Wang Chen","year":"2023","unstructured":"Chen Wang, Jialin Qiao, Xiangdong Huang, Shaoxu Song, Haonan Hou, Tian Jiang, Lei Rui, Jianmin Wang, and Jiaguang Sun. 2023. Apache IoTDB: A Time Series Database for IoT Applications. Proceedings of the ACM on Management of Data 1, 2 (2023), 1--27.","journal-title":"Proceedings of the ACM on Management of Data"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.14778\/3565816.3565829"},{"key":"e_1_2_1_38_1","volume-title":"Application of KKS Coding and QR Code Technology in Transmission Asset Management. In 2021 IEEE 2nd China International Youth Conference on Electrical Engineering (CIYCEE). IEEE, 1--6.","author":"Wang Zhenhua","year":"2021","unstructured":"Zhenhua Wang, Huikun Pei, Xiaomeng Zhang, Chenghao Wang, Xi Chen, and Te Zhou. 2021. Application of KKS Coding and QR Code Technology in Transmission Asset Management. In 2021 IEEE 2nd China International Youth Conference on Electrical Engineering (CIYCEE). IEEE, 1--6."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.14778\/3547305.3547319"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2020408.2020462"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1869790.1869807"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.14778\/3626292.3626298"},{"key":"e_1_2_1_43_1","first-page":"32","article-title":"GeoLife: A Collaborative Social Networking Service among User, Location and Trajectory","volume":"33","author":"Zheng Yu","year":"2010","unstructured":"Yu Zheng, Xing Xie, and Wei-Ying Ma. 2010. GeoLife: A Collaborative Social Networking Service among User, Location and Trajectory. IEEE Data Eng. Bull. 33, 2 (2010), 32--39. http:\/\/sites.computer.org\/debull\/A10june\/geolife.pdf","journal-title":"IEEE Data Eng. Bull."}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3685800.3685827","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,31]],"date-time":"2024-12-31T05:32:53Z","timestamp":1735623173000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3685800.3685827"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8]]},"references-count":43,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2024,8]]}},"alternative-id":["10.14778\/3685800.3685827"],"URL":"https:\/\/doi.org\/10.14778\/3685800.3685827","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2024,8]]},"assertion":[{"value":"2024-11-08","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}