{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,27]],"date-time":"2025-12-27T07:25:55Z","timestamp":1766820355936},"reference-count":20,"publisher":"Association for Computing Machinery (ACM)","issue":"12","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2021,7]]},"abstract":"<jats:p>The rapid emergence of cloud data warehouses like Google BigQuery has redefined the landscape of data analytics. With the growth of data volumes, such systems need to scale to hundreds of EiB of data in the near future. This growth is accompanied by an increase in the number of objects stored and the amount of metadata such systems must manage. Traditionally, Big Data systems have tried to reduce the amount of metadata in order to scale the system, often compromising query performance. In Google BigQuery, we built a metadata management system that demonstrates that massive scale can be achieved without such tradeoffs. We recognized the benefits that fine grained metadata provides for query processing and we built a metadata system to manage it effectively. We use the same distributed query processing and data management techniques that we use for managing data to handle Big metadata. Today, BigQuery uses these techniques to support queries over billions of objects and their metadata.<\/jats:p>","DOI":"10.14778\/3476311.3476385","type":"journal-article","created":{"date-parts":[[2021,10,28]],"date-time":"2021-10-28T22:48:43Z","timestamp":1635461323000},"page":"3083-3095","source":"Crossref","is-referenced-by-count":16,"title":["Big metadata"],"prefix":"10.14778","volume":"14","author":[{"given":"Pavan","family":"Edara","sequence":"first","affiliation":[{"name":"Google LLC"}]},{"given":"Mosha","family":"Pasumansky","sequence":"additional","affiliation":[{"name":"Google LLC"}]}],"member":"320","published-online":{"date-parts":[[2021,10,28]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1142473.1142548"},{"key":"e_1_2_1_2_1","unstructured":"Sunil Agarwal. 2017. Columnstore Index Performance: Rowgroup Elimination. https:\/\/techcommunity.microsoft.com\/t5\/sql-server\/columnstore-index-performance-rowgroup-elimination\/ba-p\/385034  Sunil Agarwal. 2017. Columnstore Index Performance: Rowgroup Elimination . https:\/\/techcommunity.microsoft.com\/t5\/sql-server\/columnstore-index-performance-rowgroup-elimination\/ba-p\/385034"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.14778\/3415478.3415560"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3314045"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.5555\/1298455.1298475"},{"key":"e_1_2_1_6_1","unstructured":"Zach Christopherson. 2016. Amazon Redshift Engineering's Advanced Table Design Playbook: Compound and Interleaved Sort Keys. https:\/\/amzn.to\/3qXXVpq  Zach Christopherson. 2016. Amazon Redshift Engineering's Advanced Table Design Playbook: Compound and Interleaved Sort Keys . https:\/\/amzn.to\/3qXXVpq"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2903741"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2742795"},{"key":"e_1_2_1_9_1","unstructured":"HMS 2020. Hive Metastore (HMS). https:\/\/docs.cloudera.com\/documentation\/enterprise\/latest\/topics\/cdh_ig_hms.html  HMS 2020. Hive Metastore (HMS) . https:\/\/docs.cloudera.com\/documentation\/enterprise\/latest\/topics\/cdh_ig_hms.html"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/119995.115835"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2008.4497486"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.14778\/2367502.2367518"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.14778\/3415478.3415568"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.14778\/1920841.1920886"},{"key":"e_1_2_1_15_1","unstructured":"Cade Metz. 2012. Google Remakes Online Empire with 'Colossus'.  Cade Metz. 2012. Google Remakes Online Empire with 'Colossus' ."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.5555\/645924.671173"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/s002360050048"},{"key":"e_1_2_1_18_1","volume-title":"Google Cloud Blog","author":"Pasumansky Mosha","year":"2016"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.14778\/2536222.2536233"},{"key":"e_1_2_1_20_1","unstructured":"S2 Cells [n.d.]. S2 Cells. https:\/\/s2geometry.io\/devguide\/s2cell_hierarchy.html  S2 Cells [n.d.]. S2 Cells . https:\/\/s2geometry.io\/devguide\/s2cell_hierarchy.html"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3476311.3476385","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T11:38:51Z","timestamp":1672227531000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3476311.3476385"}},"subtitle":["when metadata is big data"],"short-title":[],"issued":{"date-parts":[[2021,7]]},"references-count":20,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2021,7]]}},"alternative-id":["10.14778\/3476311.3476385"],"URL":"https:\/\/doi.org\/10.14778\/3476311.3476385","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2021,7]]}}}