{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T03:06:04Z","timestamp":1787022364907,"version":"build-2736575974"},"reference-count":52,"publisher":"Association for Computing Machinery (ACM)","issue":"11","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2018,7]]},"abstract":"<jats:p>\n                    Today's rapidly growing document volumes pose pressing challenges to modern document analytics, in both space usage and processing time. In this work, we propose the concept of\n                    <jats:italic>compression-based direct processing<\/jats:italic>\n                    to alleviate issues in both dimensions. The main idea is to enable direct document analytics on compressed data. We present how the concept can be materialized on Sequitur, a compression algorithm that produces hierarchical grammar-like representations. We discuss the major challenges in applying the idea to various document analytics tasks, and reveal a set of guidelines and also assistant software modules for developers to effectively apply\n                    <jats:italic>compression-based direct processing<\/jats:italic>\n                    . Experiments show that our proposed techniques save 90.8% storage space and 77.5% memory usage, while speeding up data processing significantly, i.e., by 1.6X on sequential systems, and 2.2X on distributed clusters, on average.\n                  <\/jats:p>","DOI":"10.14778\/3236187.3236203","type":"journal-article","created":{"date-parts":[[2018,9,10]],"date-time":"2018-09-10T08:12:28Z","timestamp":1536567148000},"page":"1522-1535","source":"Crossref","is-referenced-by-count":53,"title":["Efficient document analytics on compressed data"],"prefix":"10.14778","volume":"11","author":[{"given":"Feng","family":"Zhang","sequence":"first","affiliation":[{"name":"Renmin University of China, MOE and Tsinghua University, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jidong","family":"Zhai","sequence":"additional","affiliation":[{"name":"Tsinghua University, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xipeng","family":"Shen","sequence":"additional","affiliation":[{"name":"North Carolina State University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Onur","family":"Mutlu","sequence":"additional","affiliation":[{"name":"ETH Z\u00fcrich, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wenguang","family":"Chen","sequence":"additional","affiliation":[{"name":"Tsinghua University, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2018,7]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"https:\/\/code.google.com\/archive\/p\/cpp-btree\/","year":"2017","unstructured":"C++ B-tree. https:\/\/code.google.com\/archive\/p\/cpp-btree\/ , 2017 . C++ B-tree. https:\/\/code.google.com\/archive\/p\/cpp-btree\/, 2017."},{"key":"e_1_2_1_2_1","volume-title":"https:\/\/dumps.wikimedia.org\/enwiki\/","author":"HTML","year":"2017","unstructured":"Wikipedia HTML data dumps. https:\/\/dumps.wikimedia.org\/enwiki\/ , 2017 . Wikipedia HTML data dumps. https:\/\/dumps.wikimedia.org\/enwiki\/, 2017."},{"key":"e_1_2_1_3_1","volume-title":"USENIX Symposium on Networked Systems Design and Implementation","author":"Agarwal R.","year":"2015","unstructured":"R. Agarwal , A. Khandelwal , and I. Stoica . Succinct: Enabling queries on compressed data . In USENIX Symposium on Networked Systems Design and Implementation , 2015 . R. Agarwal, A. Khandelwal, and I. Stoica. Succinct: Enabling queries on compressed data. In USENIX Symposium on Networked Systems Design and Implementation, 2015."},{"key":"e_1_2_1_4_1","volume-title":"PUMA: Purdue MapReduce Benchmarks Suite","author":"Ahmad F.","year":"2012","unstructured":"F. Ahmad , S. Lee , M. Thottethodi , and T. Vijaykumar . PUMA: Purdue MapReduce Benchmarks Suite . 2012 . F. Ahmad, S. Lee, M. Thottethodi, and T. Vijaykumar. PUMA: Purdue MapReduce Benchmarks Suite. 2012."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1367497.1367673"},{"key":"e_1_2_1_6_1","volume-title":"HDFS architecture guide. HADOOP APACHE PROJECT http:\/\/hadoop. apache. org\/common\/docs\/current\/hdfs design. pdf","author":"Borthakur D.","year":"2008","unstructured":"D. Borthakur . HDFS architecture guide. HADOOP APACHE PROJECT http:\/\/hadoop. apache. org\/common\/docs\/current\/hdfs design. pdf , 2008 . D. Borthakur. HDFS architecture guide. HADOOP APACHE PROJECT http:\/\/hadoop. apache. org\/common\/docs\/current\/hdfs design. pdf, 2008."},{"key":"e_1_2_1_7_1","volume-title":"A block-sorting lossless data compression algorithm","author":"Burrows M.","year":"1994","unstructured":"M. Burrows and D. J. Wheeler . A block-sorting lossless data compression algorithm . 1994 . M. Burrows and D. J. Wheeler. A block-sorting lossless data compression algorithm. 1994."},{"key":"e_1_2_1_8_1","volume-title":"AIP Conference Proceedings","author":"Cai Q. Q.","year":"2017","unstructured":"Q. Q. Cai , H. G. Cui , and H. Tang . Provenance graph query method based on double layer index structure . In AIP Conference Proceedings , 2017 . Q. Q. Cai, H. G. Cui, and H. Tang. Provenance graph query method based on double layer index structure. In AIP Conference Proceedings, 2017."},{"key":"e_1_2_1_9_1","volume-title":"Bulletin of the IEEE Computer Society Technical Committee on Data Engineering","author":"Carbone P.","year":"2015","unstructured":"P. Carbone , A. Katsifodimos , S. Ewen , V. Markl , S. Haridi , and K. Tzoumas . Apache Flink: Stream and Batch Processing in a Single Engine . Bulletin of the IEEE Computer Society Technical Committee on Data Engineering , 2015 . P. Carbone, A. Katsifodimos, S. Ewen, V. Markl, S. Haridi, and K. Tzoumas. Apache Flink: Stream and Batch Processing in a Single Engine. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, 2015."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/378795.378840"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/512529.512554"},{"key":"e_1_2_1_12_1","volume-title":"Introduction to algorithms","author":"Cormen T. H.","year":"2009","unstructured":"T. H. Cormen . Introduction to algorithms . MIT press , 2009 . T. H. Cormen. Introduction to algorithms. MIT press, 2009."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/96749.98245"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3062341.3062377"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1412228.1455268"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1082036.1082039"},{"key":"e_1_2_1_17_1","volume-title":"Symposium on Operating Systems Design and Implementation","author":"Gonzalez J. E.","year":"2012","unstructured":"J. E. Gonzalez , Y. Low , H. Gu , D. Bickson , and C. Guestrin . PowerGraph: Distributed graph-parallel computation on natural graphs . In Symposium on Operating Systems Design and Implementation , 2012 . J. E. Gonzalez, Y. Low, H. Gu, D. Bickson, and C. Guestrin. PowerGraph: Distributed graph-parallel computation on natural graphs. In Symposium on Operating Systems Design and Implementation, 2012."},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the fifteenth annual ACM-SIAM symposium on Discrete algorithms","author":"Grossi R.","year":"2004","unstructured":"R. Grossi , A. Gupta , and J. S. Vitter . When indexing equals compression: Experiments with compressing suffix arrays and applications . In Proceedings of the fifteenth annual ACM-SIAM symposium on Discrete algorithms , 2004 . R. Grossi, A. Gupta, and J. S. Vitter. When indexing equals compression: Experiments with compressing suffix arrays and applications. In Proceedings of the fifteenth annual ACM-SIAM symposium on Discrete algorithms, 2004."},{"key":"e_1_2_1_19_1","volume-title":"USENIX Annual Technical Conference","author":"Guo F.","year":"2011","unstructured":"F. Guo and P. Efstathopoulos . Building a High-performance Deduplication System . In USENIX Annual Technical Conference , 2011 . F. Guo and P. Efstathopoulos. Building a High-performance Deduplication System. In USENIX Annual Technical Conference, 2011."},{"key":"e_1_2_1_20_1","volume-title":"New Frontiers in Information and Software as Services.","author":"Huang S.","year":"2011","unstructured":"S. Huang , J. Huang , J. Dai , T. Xie , and B. Huang . The HiBench benchmark suite: Characterization of the MapReduce-based data analysis . In New Frontiers in Information and Software as Services. 2011 . S. Huang, J. Huang, J. Dai, T. Xie, and B. Huang. The HiBench benchmark suite: Characterization of the MapReduce-based data analysis. In New Frontiers in Information and Software as Services. 2011."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/301618.301678"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISPASS.2005.1430568"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.5555\/776816.776854"},{"key":"e_1_2_1_24_1","volume-title":"Advances in Data Science and Classification.","author":"Lebart L.","year":"1998","unstructured":"L. Lebart . Classification problems in text analysis and information retrieval . In Advances in Data Science and Classification. 1998 . L. Lebart. Classification problems in text analysis and information retrieval. In Advances in Data Science and Classification. 1998."},{"key":"e_1_2_1_25_1","unstructured":"M. Lichman. UCI machine learning repository. http:\/\/archive.ics.uci.edu\/ml 2013.  M. Lichman. UCI machine learning repository. http:\/\/archive.ics.uci.edu\/ml 2013."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1066677.1066827"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807167.1807184"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.5555\/2851126"},{"key":"e_1_2_1_29_1","unstructured":"K. Mitsui. Information retrieval based on rank-ordered cumulative query scores calculated from weights of all keywords in an inverted index file for minimizing access to a main database 1993. US Patent 5 263 159.  K. Mitsui. Information retrieval based on rank-ordered cumulative query scores calculated from weights of all keywords in an inverted index file for minimizing access to a main database 1993. US Patent 5 263 159."},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the International Conference on Knowledge Discovery and Data Mining","author":"Monge A. E.","year":"1996","unstructured":"A. E. Monge , C. Elkan , The Field Matching Problem: Algorithms and Applications . In Proceedings of the International Conference on Knowledge Discovery and Data Mining , 1996 . A. E. Monge, C. Elkan, et al. The Field Matching Problem: Algorithms and Applications. In Proceedings of the International Conference on Knowledge Discovery and Data Mining, 1996."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/375360.375365"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.5555\/3092586"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1093\/comjnl\/40.2_and_3.103"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.5555\/1622776.1622780"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.5555\/789085.789587"},{"key":"e_1_2_1_37_1","volume-title":"Pthreads programming: A POSIX standard for better multiprocessing. \"O'Reilly Media","author":"Nichols B.","year":"1996","unstructured":"B. Nichols , D. Buttlar , and J. Farrell . Pthreads programming: A POSIX standard for better multiprocessing. \"O'Reilly Media , Inc .\", 1996 . B. Nichols, D. Buttlar, and J. Farrell. Pthreads programming: A POSIX standard for better multiprocessing. \"O'Reilly Media, Inc.\", 1996."},{"key":"e_1_2_1_38_1","volume-title":"et al. Scikit-learn: Machine learning in Python. Journal of machine learning research","author":"Pedregosa F.","year":"2011","unstructured":"F. Pedregosa , G. Varoquaux , A. Gramfort , V. Michel , B. Thirion , O. Grisel , M. Blondel , P. Prettenhofer , R. Weiss , V. Dubourg , et al. Scikit-learn: Machine learning in Python. Journal of machine learning research , 2011 . F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, et al. Scikit-learn: Machine learning in Python. Journal of machine learning research, 2011."},{"key":"e_1_2_1_39_1","volume-title":"Linguistic inquiry and word count: LIWC","author":"Pennebaker J. W.","year":"2001","unstructured":"J. W. Pennebaker , M. E. Francis , and R. J. Booth . Linguistic inquiry and word count: LIWC 2001 . Mahway : Lawrence Erlbaum Associates , 2001. J. W. Pennebaker, M. E. Francis, and R. J. Booth. Linguistic inquiry and word count: LIWC 2001. Mahway: Lawrence Erlbaum Associates, 2001."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2806416.2806424"},{"key":"e_1_2_1_41_1","volume-title":"Data Cleaning: Problems and Current Approaches","author":"Rahm E.","year":"2000","unstructured":"E. Rahm and H. H. Do . Data Cleaning: Problems and Current Approaches . 2000 . E. Rahm and H. H. Do. Data Cleaning: Problems and Current Approaches. 2000."},{"key":"e_1_2_1_42_1","volume-title":"Operating system concepts essentials","author":"Silberschatz A.","year":"2014","unstructured":"A. Silberschatz , P. B. Galvin , and G. Gagne . Operating system concepts essentials . John Wiley & Sons, Inc. , 2014 . A. Silberschatz, P. B. Galvin, and G. Gagne. Operating system concepts essentials. John Wiley & Sons, Inc., 2014."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.14778\/2535573.2488336"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/1868321.1868323"},{"key":"e_1_2_1_45_1","volume-title":"Inverted index storage structure using subindexes and large objects for tight coupling of information retrieval with database management systems","author":"Whang K.-Y.","year":"2002","unstructured":"K.-Y. Whang , B.-K. Park , W.-S. Han , and Y.-K. Lee . Inverted index storage structure using subindexes and large objects for tight coupling of information retrieval with database management systems , 2002 . US Patent 6,349,308. K.-Y. Whang, B.-K. Park, W.-S. Han, and Y.-K. Lee. Inverted index storage structure using subindexes and large objects for tight coupling of information retrieval with database management systems, 2002. US Patent 6,349,308."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484425.2484427"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/1526709.1526764"},{"key":"e_1_2_1_48_1","volume-title":"HotCloud","author":"Zaharia M.","year":"2010","unstructured":"M. Zaharia , M. Chowdhury , M. J. Franklin , S. Shenker , and I. Stoica . Spark: cluster computing with working sets . HotCloud , 2010 . M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica. Spark: cluster computing with working sets. HotCloud, 2010."},{"key":"e_1_2_1_49_1","volume-title":"Lexical acquisition: exploiting on-line resources to build a lexicon","author":"Zernik U.","year":"1991","unstructured":"U. Zernik . Lexical acquisition: exploiting on-line resources to build a lexicon . Psychology Press , 1991 . U. Zernik. Lexical acquisition: exploiting on-line resources to build a lexicon. Psychology Press, 1991."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/375663.375722"},{"key":"e_1_2_1_51_1","volume-title":"Potential of a Method for Text Analytics Directly on Compressed Data. Technical report","author":"Zhang F.","year":"2017","unstructured":"F. Zhang , J. Zhai , X. Shen , and O. Mutlu . Potential of a Method for Text Analytics Directly on Compressed Data. Technical report , TR-2017-4, Computer Science Department, North Carolina State University , 11 2017 . F. Zhang, J. Zhai, X. Shen, and O. Mutlu. Potential of a Method for Text Analytics Directly on Compressed Data. Technical report, TR-2017-4, Computer Science Department, North Carolina State University, 11 2017."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3205289.3205325"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.1977.1055714"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3236187.3236203","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T04:48:01Z","timestamp":1672202881000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3236187.3236203"}},"subtitle":["method, challenges, algorithms, insights"],"short-title":[],"issued":{"date-parts":[[2018,7]]},"references-count":52,"journal-issue":{"issue":"11","published-print":{"date-parts":[[2018,7]]}},"alternative-id":["10.14778\/3236187.3236203"],"URL":"https:\/\/doi.org\/10.14778\/3236187.3236203","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2018,7]]}}}