{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T03:06:04Z","timestamp":1787022364081,"version":"build-2736575974"},"reference-count":60,"publisher":"Association for Computing Machinery (ACM)","issue":"6","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2019,2]]},"abstract":"<jats:p>Storage and memory systems for modern data analytics are heavily layered, managing shared persistent data, cached data, and nonshared execution data in separate systems such as a distributed file system like HDFS, an in-memory file system like Alluxio, and a computation framework like Spark. Such layering introduces significant performance and management costs. In this paper we propose a single system called Pangea that can manage all data---both intermediate and long-lived data, and their buffer\/caching, data placement optimization, and failure recovery---all in one monolithic distributed storage system, without any layering. We present a detailed performance evaluation of Pangea and show that its performance compares favorably with several widely used layered systems such as Spark.<\/jats:p>","DOI":"10.14778\/3311880.3311885","type":"journal-article","created":{"date-parts":[[2019,3,26]],"date-time":"2019-03-26T16:22:15Z","timestamp":1553617335000},"page":"681-694","source":"Crossref","is-referenced-by-count":11,"title":["Pangea"],"prefix":"10.14778","volume":"12","author":[{"given":"Jia","family":"Zou","sequence":"first","affiliation":[{"name":"Rice University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Arun","family":"Iyengar","sequence":"additional","affiliation":[{"name":"IBM T. J. Watson Research Center"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chris","family":"Jermaine","sequence":"additional","affiliation":[{"name":"Rice University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2019,2]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Amazon simple storage system. https:\/\/aws.amazon.com\/s3.  Amazon simple storage system. https:\/\/aws.amazon.com\/s3."},{"key":"e_1_2_1_2_1","unstructured":"Apache ignite. https:\/\/ignite.apache.org.  Apache ignite. https:\/\/ignite.apache.org."},{"key":"e_1_2_1_3_1","unstructured":"Google cloud storage. https:\/\/cloud.google.com\/storage.  Google cloud storage. https:\/\/cloud.google.com\/storage."},{"key":"e_1_2_1_4_1","unstructured":"Hash table benchmark. http:\/\/incise.org\/hash-table-benchmarks.html.  Hash table benchmark. http:\/\/incise.org\/hash-table-benchmarks.html."},{"key":"e_1_2_1_5_1","unstructured":"Project tungsten: Bringing spark closer to bare metal. https:\/\/databricks.com\/blog\/2015\/04\/28\/project-tungsten-bringing-spark-closer-to-bare-metal.html.  Project tungsten: Bringing spark closer to bare metal. https:\/\/databricks.com\/blog\/2015\/04\/28\/project-tungsten-bringing-spark-closer-to-bare-metal.html."},{"key":"e_1_2_1_6_1","unstructured":"Why enterprises of different sizes are adopting 'fast data' with apache spark. https:\/\/www.lightbend.com\/blog\/why-enterprises-of-different-sizes-are-adopting-fast-data-with-apache-spark.  Why enterprises of different sizes are adopting 'fast data' with apache spark. https:\/\/www.lightbend.com\/blog\/why-enterprises-of-different-sizes-are-adopting-fast-data-with-apache-spark."},{"key":"e_1_2_1_7_1","unstructured":"M. Abadi A. Agarwal P. Barham E. Brevdo Z. Chen C. Citro G. S. Corrado A. Davis J. Dean M. Devin et al. Tensorflow: Large-scale machine learning on heterogeneous systems 2015. Software available from tensorflow. org.  M. Abadi A. Agarwal P. Barham E. Brevdo Z. Chen C. Citro G. S. Corrado A. Davis J. Dean M. Devin et al. Tensorflow: Large-scale machine learning on heterogeneous systems 2015. Software available from tensorflow. org."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1007568.1007609"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-014-0357-y"},{"key":"e_1_2_1_10_1","first-page":"20","volume-title":"Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation","author":"Ananthanarayanan G.","year":"2012"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2742797"},{"key":"e_1_2_1_12_1","unstructured":"J. Arnold. Openstack swift: Using administering and developing for swift object storage. \" O'Reilly Media Inc.\" 2014.   J. Arnold. Openstack swift: Using administering and developing for swift object storage. \" O'Reilly Media Inc.\" 2014."},{"key":"e_1_2_1_13_1","first-page":"365","volume-title":"NSDI","volume":"4","author":"Bent J.","year":"2004"},{"key":"e_1_2_1_14_1","unstructured":"D. Borthakur. Hdfs architecture guide. HADOOP APACHE PROJECT http:\/\/hadoop.apache.org\/common\/docs\/current\/hdfsdesign.pdf 2008.  D. Borthakur. Hdfs architecture guide. HADOOP APACHE PROJECT http:\/\/hadoop.apache.org\/common\/docs\/current\/hdfsdesign.pdf 2008."},{"key":"e_1_2_1_15_1","unstructured":"D. P. Bovet and M. Cesati. Understanding the Linux kernel. \" O'Reilly Media Inc.\" 2005.   D. P. Bovet and M. Cesati. Understanding the Linux kernel. \" O'Reilly Media Inc.\" 2005."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2043556.2043571"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/235543.235544"},{"key":"e_1_2_1_18_1","first-page":"193","volume-title":"Usenix symposium on internet technologies and systems","volume":"12","author":"Cao P.","year":"1997"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.14778\/1454159.1454166"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.14778\/2367502.2367519"},{"key":"e_1_2_1_21_1","doi-asserted-by":"crossref","unstructured":"H.-T. Chou and D. J. DeWitt. An evaluation of buffer management strategies for relational database systems. Algorithmica 1(1--4):311--336 1986.  H.-T. Chou and D. J. DeWitt. An evaluation of buffer management strategies for relational database systems. Algorithmica 1(1--4):311--336 1986.","DOI":"10.1007\/BF01840450"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.14778\/2824032.2824045"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.14778\/1920841.1920908"},{"key":"e_1_2_1_24_1","unstructured":"D. Ellard E. Thereska G. R. Ganger M. I. Seltzer etal Attribute-based prediction of file properties. 2003.  D. Ellard E. Thereska G. R. Ganger M. I. Seltzer et al. Attribute-based prediction of file properties. 2003."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.14778\/2002938.2002943"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1137\/0207025"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.5555\/1012889.1012894"},{"key":"e_1_2_1_28_1","volume-title":"Boston University Computer Science Department","author":"Fonseca R.","year":"2002"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/INFOCOM.2015.7218613"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1165389.945450"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1147\/JRD.2011.2165682"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/1816038.1815971"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3190656"},{"key":"e_1_2_1_34_1","first-page":"117","volume-title":"OSDI","author":"Jyothi S. A.","year":"2016"},{"key":"e_1_2_1_35_1","volume-title":"Computer applications","author":"Kleinrock L.","year":"1976"},{"key":"e_1_2_1_36_1","unstructured":"M. Kornacker and J. Erickson. Cloudera Impala: Real time queries in apache hadoop for real. http:\/\/blog.cloudera.com\/blog\/2012\/10\/cloudera-impala-real-time-queries-in-apache-hadoop-for-real 2012.  M. Kornacker and J. Erickson. Cloudera Impala: Real time queries in apache hadoop for real. http:\/\/blog.cloudera.com\/blog\/2012\/10\/cloudera-impala-real-time-queries-in-apache-hadoop-for-real 2012."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/TC.2001.970573"},{"key":"e_1_2_1_38_1","unstructured":"H. Li. Alluxio: A virtual distributed file system. 2018.  H. Li. Alluxio: A virtual distributed file system. 2018."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2670979.2670985"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/234215.234473"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.14778\/2994509.2994513"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.5555\/1009383.1009826"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.5555\/1078026.1078408"},{"key":"e_1_2_1_44_1","unstructured":"A. Morton. Usermode pagecache control: fadvise ().  A. Morton. Usermode pagecache control: fadvise ()."},{"key":"e_1_2_1_45_1","first-page":"385","volume-title":"NSDI","author":"Nishtala R.","year":"2013"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/170036.170081"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/332799.332895"},{"key":"e_1_2_1_48_1","unstructured":"F. Pedregosa G. Varoquaux A. Gramfort V. Michel B. Thirion O. Grisel M. Blondel P. Prettenhofer R. Weiss V. Dubourg et al. Scikit-learn: Machine learning in python. Journal of machine learning research 12(Oct):2825--2830 2011.   F. Pedregosa G. Varoquaux A. Gramfort V. Michel B. Thirion O. Grisel M. Blondel P. Prettenhofer R. Weiss V. Dubourg et al. Scikit-learn: Machine learning in python. Journal of machine learning research 12(Oct):2825--2830 2011."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/564691.564757"},{"key":"e_1_2_1_50_1","unstructured":"S. Sanfilippo and P. Noordhuis. Redis 2009.  S. Sanfilippo and P. Noordhuis. Redis 2009."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.14778\/2535573.2488339"},{"key":"e_1_2_1_52_1","first-page":"553","volume-title":"Proceedings of the 31st international conference on Very large data bases","author":"Stonebraker M.","year":"2005"},{"key":"e_1_2_1_53_1","first-page":"307","volume-title":"Proceedings of the 7th symposium on Operating systems design and implementation","author":"Weil S. A.","year":"2006"},{"key":"e_1_2_1_54_1","unstructured":"T. White. Hadoop: The Definitive Guide. O'Reilly Media 2012.   T. White. Hadoop: The Definitive Guide. O'Reilly Media 2012."},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/2508148.2485965"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF01189992"},{"key":"e_1_2_1_57_1","first-page":"2","volume-title":"NSDI","author":"Zaharia M.","year":"2012"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/2213836.2213839"},{"key":"e_1_2_1_59_1","first-page":"91","volume-title":"USENIX Annual Technical Conference, General Track","author":"Zhou Y.","year":"2001"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3196933"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3311880.3311885","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T04:18:43Z","timestamp":1672201123000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3311880.3311885"}},"subtitle":["monolithic distributed storage for data analytics"],"short-title":[],"issued":{"date-parts":[[2019,2]]},"references-count":60,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2019,2]]}},"alternative-id":["10.14778\/3311880.3311885"],"URL":"https:\/\/doi.org\/10.14778\/3311880.3311885","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2019,2]]}}}