{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,7]],"date-time":"2026-04-07T21:12:14Z","timestamp":1775596334702,"version":"3.50.1"},"reference-count":39,"publisher":"Association for Computing Machinery (ACM)","issue":"1","funder":[{"name":"National Key Research and Development Plan","award":["2025ZD1601701, 2024YFB3311901, 2021YFB3300500"],"award-info":[{"award-number":["2025ZD1601701, 2024YFB3311901, 2021YFB3300500"]}]},{"name":"National Natural Science Foundation of China","award":["62232005, 92267203, 62021002"],"award-info":[{"award-number":["62232005, 92267203, 62021002"]}]},{"name":"State Grid Ningxia Electric Power Co. Science and Technology Project","award":["SGNXYX00SCJS2400058"],"award-info":[{"award-number":["SGNXYX00SCJS2400058"]}]},{"name":"Beijing National Research Center For Information Science And Technology","award":["BNR2025RC01011"],"award-info":[{"award-number":["BNR2025RC01011"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Manag. Data"],"published-print":{"date-parts":[[2026,4,2]]},"abstract":"<jats:p>LZ4 is a compression algorithm widely adopted in many database systems, which surprisingly has no support for directly querying the compressed data. Existing systems rely on full decompression for query processing, leading to increased query latency. Moreover, the LZ4 compression algorithm has issues like long match dependency, which reduces both compression effectiveness and the efficiency of compressed-data query processing. In this paper, (1) we propose LZV, a compression algorithm that employs a variable-length hash mechanism to identify maximal matches, significantly improving the compression ratio and reducing the compressed-data query overhead. (2) We propose compressed index search method that leverages auxiliary structure to efficiently query compressed data in LZ4 format directly. (3) Leveraging the ordered, quasi-uniformly spaced nature of the key column, we introduce compressed key search method that integrates prediction with binary search to retrieve the corresponding index for a given key efficiently. Finally, we implement the compressed-data query pipeline in Apache TsFile, an open-source KV storage system. Experimental results show that our approach significantly improves compression effectiveness and query efficiency.<\/jats:p>","DOI":"10.1145\/3786660","type":"journal-article","created":{"date-parts":[[2026,4,7]],"date-time":"2026-04-07T17:54:13Z","timestamp":1775584453000},"page":"1-26","source":"Crossref","is-referenced-by-count":0,"title":["Improving LZ4 for Effective Compression and Efficient Query"],"prefix":"10.1145","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-6727-9143","authenticated-orcid":false,"given":"Zhiheng","family":"Liu","sequence":"first","affiliation":[{"name":"Tsinghua University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9503-2755","authenticated-orcid":false,"given":"Shaoxu","family":"Song","sequence":"additional","affiliation":[{"name":"Tsinghua University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2026,4,7]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"2025. Source Code and Datasets. https:\/\/github.com\/liuzhiheng20\/LZV. Accessed: 2025-07--12."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1376616.1376712"},{"key":"e_1_2_1_3_1","unstructured":"Apache Cassandra Documentation. 2023. Compression. https:\/\/cassandra.apache.org\/doc\/latest\/cassandra\/managing\/operating\/compression.html. Accessed: 2025-07-03."},{"key":"e_1_2_1_4_1","unstructured":"Apache IoTDB Documentation. 2025. Encoding and Compression\u2014Apache IoTDB. https:\/\/iotdb.apache.org\/UserGuide\/latest-Table\/Technical-Insider\/Encoding-and-Compression.html. Accessed: 2025-07-03."},{"key":"e_1_2_1_5_1","unstructured":"Apache TsFile. n.d.. Apache TsFile GitHub Repository. https:\/\/github.com\/apache\/tsfile. Accessed: 2025-07-07."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.24432\/C5DW43"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICECS.2015.7440278"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/RECONFIG48160.2019.8994794"},{"key":"e_1_2_1_9_1","unstructured":"ClickHouse Documentation. 2024. Compression Modes \u2014 ClickHouse Documentation. https:\/\/clickhouse.com\/docs\/en\/data-compression\/compression-modes. Accessed: 2025-07-03."},{"key":"e_1_2_1_10_1","volume-title":"Real Time Data Compression: LZ4 Explained. https:\/\/fastcompression.blogspot.com\/2011\/05\/lz4-explained.html [Online","author":"Collet Yann","year":"2025","unstructured":"Yann Collet. 2011. Real Time Data Compression: LZ4 Explained. https:\/\/fastcompression.blogspot.com\/2011\/05\/lz4-explained.html [Online; accessed 28-June-2025]."},{"key":"e_1_2_1_11_1","volume-title":"A framework for the analysis of unevenly spaced time series data. Preprint. Available at: http:\/\/www. eckner. com\/papers\/unevenly_spaced_time_series_analysis","author":"Eckner Andreas","year":"2012","unstructured":"Andreas Eckner. 2012. A framework for the analysis of unevenly spaced time series data. Preprint. Available at: http:\/\/www. eckner. com\/papers\/unevenly_spaced_time_series_analysis (2012), 93."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/DCC.2013.21"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/S11280-019-00667-Z"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","unstructured":"Hogue and John. 2019. Metro Interstate Traffic Volume. UCI Machine Learning Repository. DOI: https:\/\/doi.org\/10.24432\/C5X60B.","DOI":"10.24432\/C5X60B"},{"key":"e_1_2_1_15_1","volume-title":"Snappy: A Fast Compression Format. https:\/\/github.com\/google\/snappy. Accessed: 2025-04--28.","author":"Google Inc.","year":"2023","unstructured":"Google Inc. 2023. Snappy: A Fast Compression Format. https:\/\/github.com\/google\/snappy. Accessed: 2025-04--28."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.2974489"},{"key":"e_1_2_1_17_1","volume-title":"The Art of Computer Programming, Volume 3: Sorting and Searching","author":"Knuth Donald E.","unstructured":"Donald E. Knuth. 1998. The Art of Computer Programming, Volume 3: Sorting and Searching (2nd ed.). Addison-Wesley.","edition":"2"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3196909"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/DCC.2010.29"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1007\/978--3--642--21458--5_6"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3639320"},{"key":"e_1_2_1_22_1","unstructured":"LZ4. n.d.. LZ4 - Extremely Fast Compression. https:\/\/lz4.org\/ Accessed: 2025-07--16."},{"key":"e_1_2_1_23_1","first-page":"10","article-title":"LZW data compression","volume":"14","author":"Nelson Mark R.","year":"1989","unstructured":"Mark R. Nelson. 1989. LZW data compression. Dr. Dobb's J. 14, 10 (Oct. 1989), 29--36.","journal-title":"Dr. Dobb's J."},{"key":"e_1_2_1_24_1","unstructured":"Oracle Corporation. 2024. Class HashMap (Java Platform SE 8). https:\/\/docs.oracle.com\/javase\/8\/docs\/api\/java\/util\/HashMap.html. Accessed: 2025--10--14."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.14778\/2824032.2824078"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1007\/S41019-024-00273--8"},{"key":"e_1_2_1_27_1","unstructured":"RocksDB Documentation. 2022. Compression (RocksDB Wiki). https:\/\/github.com\/facebook\/rocksdb\/wiki\/Compression. Accessed: 2025-07-03."},{"key":"e_1_2_1_28_1","volume-title":"Data Compression: The Complete Reference","author":"Salomon David","year":"2004","unstructured":"David Salomon. 2004. Data Compression: The Complete Reference, 3rd Edition. Springer. http:\/\/www.davidsalomon.name\/DC3advertis\/DComp3Ad.html","edition":"3"},{"key":"e_1_2_1_29_1","volume-title":"Predicting in-hospital mortality of icu patients: The physionet\/computing in cardiology challenge","author":"Silva Ikaro","year":"2012","unstructured":"Ikaro Silva, George Moody, Daniel J Scott, Leo A Celi, and Roger G Mark. 2012. Predicting in-hospital mortality of icu patients: The physionet\/computing in cardiology challenge 2012. In 2012 computing in cardiology. IEEE, 245--248."},{"key":"e_1_2_1_30_1","unstructured":"LanceDB Team. 2025. Columnar File Readers In-Depth: Compression Transparency. https:\/\/www.lancedb.com\/blog\/blog\/columnar-file-readers-in-depth-compression-transparency\/. Accessed: 2025-07-06."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.24432\/C58C86"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","unstructured":"Vahdat Mehrnoosh Oneto Luca Anguita Davide Funk Mathias Rauterberg and Matthias. 2015. Educational Process Mining (EPM): A Learning Analytics Data Set. UCI Machine Learning Repository. DOI: https:\/\/doi.org\/10.24432\/C5NP5K.","DOI":"10.24432\/C5NP5K"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","unstructured":"Vito and Saverio. 2008. Air Quality. UCI Machine Learning Repository. DOI: https:\/\/doi.org\/10.24432\/C59K5F.","DOI":"10.24432\/C59K5F"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3589775"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.5220\/0006235400360046"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.14778\/3626292.3626298"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.14778\/3685800.3685827"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.1977.1055714"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.1978.1055934"}],"container-title":["Proceedings of the ACM on Management of Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3786660","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,7]],"date-time":"2026-04-07T20:03:54Z","timestamp":1775592234000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3786660"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,2]]},"references-count":39,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2026,4,2]]}},"alternative-id":["10.1145\/3786660"],"URL":"https:\/\/doi.org\/10.1145\/3786660","relation":{},"ISSN":["2836-6573"],"issn-type":[{"value":"2836-6573","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,2]]}}}