{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,17]],"date-time":"2026-05-17T07:16:19Z","timestamp":1779002179551,"version":"3.51.4"},"reference-count":90,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2024,2,19]],"date-time":"2024-02-19T00:00:00Z","timestamp":1708300800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62025203"],"award-info":[{"award-number":["62025203"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Storage"],"published-print":{"date-parts":[[2024,5,31]]},"abstract":"<jats:p>Cloud logs can be categorized into on-line, off-line, and near-line logs based on the access frequency. Among them, near-line logs are mainly used for debugging, which means they prefer a low query latency for better user experience. Besides, the storage system for near-line logs prefers a low overall cost including the storage cost to store compressed logs, and the computation cost to compress logs and execute queries. These requirements pose challenges to achieving fast and cheap cloud log storage.<\/jats:p>\n          <jats:p>This article proposes LogGrep, the first log compression and query tool that exploits both static and runtime patterns to properly structurize and organize log data in fine-grained units. The key idea of LogGrep is \u201cvertical partitioning\u201d: it stores each log entry into multiple partitions by first parsing logs into variable vectors according to static patterns and then extracting runtime pattern(s) automatically within each variable vector. Based on such runtime patterns, LogGrep further decomposes the variable vectors into fine-grained units called \u201cCapsules\u201d and stamps each Capsule with a summary of its values. During the query process, LogGrep can avoid decompressing and scanning Capsules that cannot match the keywords, with the help of the extracted runtime patterns and the Capsule stamps. We further show that the interactive debugging can well utilize the advantages of the vertical-partitioning-based method and mitigate its weaknesses as well. To this end, LogGrep integrates incremental locating and partial reconstruction to mitigate the read amplification incurred by vertical-partitioning-based method.<\/jats:p>\n          <jats:p>We evaluate LogGrep on 37 cloud logs from the production environment of Alibaba Cloud and the public datasets. The results show that LogGrep can reduce the query latency and the overall cost by an order of magnitude compared with state-of-the-art works. Such results have confirmed that it is worthwhile applying a more sophisticated vertical-partitioning-based method to accelerate queries on compressed cloud logs.<\/jats:p>","DOI":"10.1145\/3643641","type":"journal-article","created":{"date-parts":[[2024,1,29]],"date-time":"2024-01-29T12:41:58Z","timestamp":1706532118000},"page":"1-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Exploiting Data-pattern-aware Vertical Partitioning to Achieve Fast and Low-cost Cloud Log Storage"],"prefix":"10.1145","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0688-6370","authenticated-orcid":false,"given":"Junyu","family":"Wei","sequence":"first","affiliation":[{"name":"Tsinghua University, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3480-5902","authenticated-orcid":false,"given":"Guangyan","family":"Zhang","sequence":"additional","affiliation":[{"name":"Tsinghua University, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6273-075X","authenticated-orcid":false,"given":"Junchao","family":"Chen","sequence":"additional","affiliation":[{"name":"Tsinghua University, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9721-4923","authenticated-orcid":false,"given":"Yang","family":"Wang","sequence":"additional","affiliation":[{"name":"The Ohio State University, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-3026-2278","authenticated-orcid":false,"given":"Weimin","family":"Zheng","sequence":"additional","affiliation":[{"name":"Tsinghua University, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3334-1309","authenticated-orcid":false,"given":"Tingtao","family":"Sun","sequence":"additional","affiliation":[{"name":"Alibaba Group, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7417-5469","authenticated-orcid":false,"given":"Jiesheng","family":"Wu","sequence":"additional","affiliation":[{"name":"Alibaba Group, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7759-800X","authenticated-orcid":false,"given":"Jiangwei","family":"Jiang","sequence":"additional","affiliation":[{"name":"Alibaba Group, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,2,19]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.1561\/1900000024"},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/1142473.1142548"},{"key":"e_1_3_1_4_2","first-page":"337","volume-title":"NSDI\u201915 Proceedings of the 12th USENIX Conference on Networked Systems Design and Implementation","author":"Agarwal Rachit","year":"2015","unstructured":"Rachit Agarwal, Anurag Khandelwal, and Ion Stoica. 2015. Succinct: Enabling queries on compressed data. In NSDI\u201915 Proceedings of the 12th USENIX Conference on Networked Systems Design and Implementation. 337\u2013350."},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/CSNT.2014.92"},{"key":"e_1_3_1_6_2","article-title":"Producition logs sample","author":"authors LogGrep","year":"2022","unstructured":"LogGrep authors. 2022. Producition logs sample. Retrieved from https:\/\/github.com\/THUBear-wjy\/openSample","journal-title":"R"},{"key":"e_1_3_1_7_2","unstructured":"LogGrep authors. 2023. Source code of LogGrep. Retrieved from https:\/\/github.com\/THUBear-wjy\/LogGrep"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/359842.359859"},{"key":"e_1_3_1_9_2","unstructured":"Elasticsearch B. V.2020. Elasticsearch 7.8.0. Retrieved from https:\/\/www.elastic.co\/downloads\/past-releases\/elasticsearch-7-8-0"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3448016.3457565"},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/1365815.1365816"},{"key":"e_1_3_1_12_2","first-page":"217","volume-title":"Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation","author":"Chow Michael","year":"2014","unstructured":"Michael Chow, David Meisner, Jason Flinn, Daniel Peek, and Thomas F. Wenisch. 2014. The mystery machine: End-to-end performance analysis of large-scale internet services. In Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation. USENIX Association, 217\u2013231."},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/1108792.1108814"},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCOM.1984.1096090"},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/96749.98245"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2903741"},{"key":"e_1_3_1_17_2","doi-asserted-by":"crossref","unstructured":"Peter Deutsch. 1996. DEFLATE Compressed Data Format Specification version 1.3. Retrieved from https:\/\/tools.ietf.org\/html\/rfc1951","DOI":"10.17487\/rfc1951"},{"key":"e_1_3_1_18_2","unstructured":"Linux developer community. 2020. Grep Manuel. Retrieved from https:\/\/linux.die.net\/man\/1\/grep"},{"key":"e_1_3_1_19_2","unstructured":"GNU developer group. 2019. Homepage and documentation of Tar. Retrieved from https:\/\/www.gnu.org\/software\/tar\/"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3133956.3134015"},{"key":"e_1_3_1_21_2","unstructured":"Jarek Duda. 2013. Asymmetric numeral systems as close to capacity low state entropy coders. arXiv:1311.2540. Retrieved from https:\/\/arxiv.org\/abs\/1311.2540"},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4939-0378-8_14"},{"key":"e_1_3_1_23_2","unstructured":"Facebook. 2021. z-standard compression tool. Retrieved from https:\/\/github.com\/facebook\/zstd"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2015.2489657"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10844-017-0450-y"},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/TrustCom.2016.0215"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/2790755.2790797"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0020-0255(01)00098-6"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/1328438.1328488"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3448016.3452798"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3183746"},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2014.2384497"},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/WCNC.2004.1311529"},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1137\/S0097539702402354"},{"key":"e_1_3_1_35_2","unstructured":"ElasticSearch group. 2019. Elasticsearch: Bulk Inserting Examples. Retrieved from https:\/\/queirozf.com\/entries\/elasticsearch-bulk-inserting-examples"},{"key":"e_1_3_1_36_2","unstructured":"Loghub group. 2019. Download link of public log dataset. Retrieved from https:\/\/zenodo.org\/record\/7056802#.Yxm2VexBwq2"},{"key":"e_1_3_1_37_2","unstructured":"LogArchive group. 2019. Open source code of LogArchive. Retrieved from https:\/\/github.com\/robertchristensen\/log_archive_v0"},{"key":"e_1_3_1_38_2","unstructured":"Logzip group. 2019. Open source code of Logzip. Retrieved from https:\/\/github.com\/logpai\/logzip"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/2983323.2983358"},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICWS.2017.13"},{"key":"e_1_3_1_41_2","unstructured":"Shilin He Jieming Zhu Pinjia He and Michael R. Lyu. 2020. Loghub: A large collection of system log datasets towards automated log analytics. arXiv:2008.06448. Retrieved from https:\/\/arxiv.org\/abs\/2008.06448"},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/JRPROC.1952.273898"},{"key":"e_1_3_1_43_2","unstructured":"Splunk Inc.2020. Spunk Enterprise 8.0.3. Retrieved from https:\/\/www.splunk.com\/en_us\/download\/previous-releases.html"},{"key":"e_1_3_1_44_2","unstructured":"Scalyr Inc.2021. Scalyr home page. Retrieved from https:\/\/www.scalyr.com\/"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","DOI":"10.14778\/3380750.3380761"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/QSIC.2008.50"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/DCC.1998.672136"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-024X(199911)29:13<1149::AID-SPE274>3.0.CO;2-O"},{"key":"e_1_3_1_49_2","unstructured":"Grafana Labs. 2021. Loki Documentation. Retrieved from https:grafana.com\/docs\/loki\/latest\/"},{"key":"e_1_3_1_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2882925"},{"key":"e_1_3_1_51_2","unstructured":"George Lee Jimmy Lin Chuang Liu Andrew Lorek and Dmitriy Ryaboy. 2012. The unified logging infrastructure for data analytics at Twitter. arXiv:1208.4171. Retrieved from https:\/\/arxiv.org\/abs\/1208.4171"},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3448016.3452829"},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3448016.3457277"},{"key":"e_1_3_1_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/CCGrid.2015.45"},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00085"},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2011.138"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/1557019.1557154"},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3196321.3196340"},{"key":"e_1_3_1_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/SCC.2013.73"},{"key":"e_1_3_1_60_2","doi-asserted-by":"publisher","DOI":"10.1002\/widm.53"},{"key":"e_1_3_1_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2010.5463281"},{"key":"e_1_3_1_62_2","doi-asserted-by":"publisher","DOI":"10.5555\/2228298.2228334"},{"key":"e_1_3_1_63_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-21903-5_8"},{"key":"e_1_3_1_64_2","doi-asserted-by":"publisher","DOI":"10.1109\/DSN.2015.14"},{"key":"e_1_3_1_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/3289600.3290962"},{"key":"e_1_3_1_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/3415148"},{"key":"e_1_3_1_67_2","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-51486-4_90"},{"key":"e_1_3_1_68_2","first-page":"183","volume-title":"Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201921)","author":"Rodrigues Kirk","year":"2021","unstructured":"Kirk Rodrigues, Yu Luo, and Ding Yuan. 2021. CLP: Efficient and scalable search on compressed text logs. In Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201921). 183\u2013198."},{"key":"e_1_3_1_69_2","first-page":"225","volume-title":"Proceedings of the 13th Annual ACM-SIAM Symposium on Discrete Algorithms","author":"Sadakane Kunihiko","year":"2002","unstructured":"Kunihiko Sadakane. 2002. Succinct representations of lcp information and improvements in the compressed suffix arrays. In Proceedings of the 13th Annual ACM-SIAM Symposium on Discrete Algorithms. 225\u2013232."},{"key":"e_1_3_1_70_2","volume-title":"Data Compression: The Complete Reference","author":"Salomon D.","year":"2004","unstructured":"D. Salomon. 2004. Data Compression: The Complete Reference. Springer, New York."},{"key":"e_1_3_1_71_2","unstructured":"Vijay Samuel. 2018. Monitoring anything and everything with beats at eBay. (2018)."},{"key":"e_1_3_1_72_2","volume-title":"Introduction to Data Compression","author":"Sayood Khalid","year":"2017","unstructured":"Khalid Sayood. 2017. Introduction to Data Compression. Morgan Kaufmann."},{"key":"e_1_3_1_73_2","unstructured":"Keiichi Shima. 2016. Length matters: Clustering system log messages using length of words. arXiv:1611.03213. Retrieved from https:\/\/arxiv.org\/abs\/1611.03213"},{"key":"e_1_3_1_74_2","doi-asserted-by":"publisher","DOI":"10.14778\/3025111.3025123"},{"key":"e_1_3_1_75_2","doi-asserted-by":"publisher","DOI":"10.1145\/2063576.2063690"},{"key":"e_1_3_1_76_2","doi-asserted-by":"publisher","DOI":"10.1109\/IPOM.2003.1251233"},{"key":"e_1_3_1_77_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.artmed.2014.01.002"},{"key":"e_1_3_1_78_2","doi-asserted-by":"publisher","DOI":"10.1145\/3552326.3567484"},{"key":"e_1_3_1_79_2","first-page":"249","volume-title":"Proceedings of the 19th USENIX Conference on File and Storage Technologies (FAST\u201921)","author":"Wei Junyu","year":"2021","unstructured":"Junyu Wei, Guangyan Zhang, Yang Wang, Zhiwei Liu, Zhanyang Zhu, Junchao Chen, Tingtao Sun, and Qi Zhou. 2021. On the feasibility of parser-based log compression in large-scale cloud systems. In Proceedings of the 19th USENIX Conference on File and Storage Technologies (FAST\u201921). 249\u2013262."},{"key":"e_1_3_1_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/214762.214771"},{"key":"e_1_3_1_81_2","doi-asserted-by":"publisher","DOI":"10.1145\/1629575.1629587"},{"key":"e_1_3_1_82_2","unstructured":"YScope. 2021. clp-core. Retrieved from https:\/\/github.com\/y-scope\/clp-core"},{"key":"e_1_3_1_83_2","doi-asserted-by":"publisher","DOI":"10.1145\/1736020.1736038"},{"key":"e_1_3_1_84_2","doi-asserted-by":"publisher","DOI":"10.1145\/3514221.3526130"},{"key":"e_1_3_1_85_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-020-00636-3"},{"key":"e_1_3_1_86_2","doi-asserted-by":"publisher","DOI":"10.1109\/COMPSAC.2017.137"},{"key":"e_1_3_1_87_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2010.211"},{"key":"e_1_3_1_88_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP.2019.00021"},{"key":"e_1_3_1_89_2","unstructured":"7 zip developer group. 2019. 7-zip file achiever home page. Retrieved from https:\/\/www.7-zip.org\/"},{"key":"e_1_3_1_90_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.1977.1055714"},{"key":"e_1_3_1_91_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.1977.1055714"}],"container-title":["ACM Transactions on Storage"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3643641","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3643641","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:50:28Z","timestamp":1750287028000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3643641"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,19]]},"references-count":90,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,5,31]]}},"alternative-id":["10.1145\/3643641"],"URL":"https:\/\/doi.org\/10.1145\/3643641","relation":{},"ISSN":["1553-3077","1553-3093"],"issn-type":[{"value":"1553-3077","type":"print"},{"value":"1553-3093","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,2,19]]},"assertion":[{"value":"2023-01-03","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-12-26","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-02-19","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}