{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,19]],"date-time":"2025-09-19T06:51:19Z","timestamp":1758264679811},"reference-count":33,"publisher":"Association for Computing Machinery (ACM)","issue":"14","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2016,10]]},"abstract":"<jats:p>\n            This paper introduces Quill (stands for a\n            <jats:italic>quadrillion tuples per day<\/jats:italic>\n            ), a library and distributed platform for relational and temporal analytics over large datasets in the cloud. Quill exposes a new abstraction for parallel datasets and computation, called\n            <jats:italic>ShardedStreamable<\/jats:italic>\n            . This abstraction provides the ability to express efficient distributed physical query plans that are transferable, i.e., movable from offline to real-time and vice versa. ShardedStreamable decouples incremental query logic specification, a small but rich set of data movement operations, and keying; this allows Quill to express a broad space of plans with complex querying functionality, while leveraging existing temporal libraries such as Trill. Quill's layered architecture provides a careful separation of responsibilities with independently useful components, while retaining high performance. We built Quill for the cloud, with a master-less design where a language-integrated client library directly communicates and coordinates with cloud workers using off-the-shelf distributed cloud components such as queues. Experiments on up to 400 cloud machines, and on datasets up to 1TB, find Quill to incur low overheads and outperform SparkSQL by up to orders-of-magnitude for temporal and 6\u00d7 for relational queries, while supporting a rich space of transferable, programmable, and expressive distributed physical query plans.\n          <\/jats:p>","DOI":"10.14778\/3007328.3007329","type":"journal-article","created":{"date-parts":[[2016,11,1]],"date-time":"2016-11-01T13:47:47Z","timestamp":1478008067000},"page":"1623-1634","source":"Crossref","is-referenced-by-count":7,"title":["Quill"],"prefix":"10.14778","volume":"9","author":[{"given":"Badrish","family":"Chandramouli","sequence":"first","affiliation":[{"name":"Microsoft Research"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Raul Castro","family":"Fernandez","sequence":"additional","affiliation":[{"name":"MIT"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jonathan","family":"Goldstein","sequence":"additional","affiliation":[{"name":"Microsoft Research"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ahmed","family":"Eldawy","sequence":"additional","affiliation":[{"name":"Univ. of Minnesota"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Abdul","family":"Quamar","sequence":"additional","affiliation":[{"name":"Univ. of Maryland"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2016,10]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"CIDR","author":"Abadi D. J.","year":"2005","unstructured":"D. J. Abadi The Design of the Borealis Stream Processing Engine . In CIDR , 2005 . D. J. Abadi et al. The Design of the Borealis Stream Processing Engine. In CIDR, 2005."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.14778\/2536222.2536229"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.14778\/2824032.2824076"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-014-0357-y"},{"key":"e_1_2_1_5_1","unstructured":"Apache. Calcite. http:\/\/calcite.apache.org.  Apache. Calcite. http:\/\/calcite.apache.org."},{"key":"e_1_2_1_6_1","unstructured":"Apache Kafka. http:\/\/kafka.apache.org\/.  Apache Kafka. http:\/\/kafka.apache.org\/."},{"key":"e_1_2_1_7_1","unstructured":"Apache Storm. http:\/\/storm.apache.org\/.  Apache Storm. http:\/\/storm.apache.org\/."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2742797"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/543613.543615"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807167.1807273"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2009.68"},{"key":"e_1_2_1_12_1","unstructured":"B. Chandramouli etal The Quill Distributed Analytics Library and Platform. Technical report Microsoft Research (MSR-TR-2016-25). http:\/\/aka.ms\/quill-tr.  B. Chandramouli et al. The Quill Distributed Analytics Library and Platform. Technical report Microsoft Research (MSR-TR-2016-25). http:\/\/aka.ms\/quill-tr."},{"key":"e_1_2_1_13_1","volume-title":"StreamRec: A Real-Time Recommender System","author":"Chandramouli B.","year":"2011","unstructured":"B. Chandramouli StreamRec: A Real-Time Recommender System . 2011 . B. Chandramouli et al. StreamRec: A Real-Time Recommender System. 2011."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.14778\/2556549.2556557"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.14778\/2735496.2735503"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2012.55"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2750545"},{"key":"e_1_2_1_18_1","unstructured":"Cloudera Engineering Blog. Cloudera Impala: Real-Time Queries in Apache Hadoop For Real. http:\/\/tinyurl.com\/bsrhcf7.  Cloudera Engineering Blog. Cloudera Impala: Real-Time Queries in Apache Hadoop For Real. http:\/\/tinyurl.com\/bsrhcf7."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1327452.1327492"},{"key":"e_1_2_1_20_1","unstructured":"Forcing Query Plans in SQL Server. http:\/\/aka.ms\/etgyp4.  Forcing Query Plans in SQL Server. http:\/\/aka.ms\/etgyp4."},{"key":"e_1_2_1_21_1","unstructured":"GitHub Archive. https:\/\/www.githubarchive.org\/.  GitHub Archive. https:\/\/www.githubarchive.org\/."},{"key":"e_1_2_1_22_1","first-page":"18","year":"1995","unstructured":"G. Graefe. The Cascades Framework for Query Optimization. Data Engineering Bulletin , 18 , 1995 . G. Graefe. The Cascades Framework for Query Optimization. Data Engineering Bulletin, 18, 1995.","journal-title":"G. Graefe. The Cascades Framework for Query Optimization. Data Engineering Bulletin"},{"key":"e_1_2_1_23_1","volume-title":"CIDR","author":"Kornacker M.","year":"2015","unstructured":"M. Kornacker : A modern, open-source SQL engine for Hadoop . In CIDR , 2015 . M. Kornacker et al. Impala: A modern, open-source SQL engine for Hadoop. In CIDR, 2015."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/1058150.1058158"},{"key":"e_1_2_1_25_1","unstructured":"N. Marz. How to beat the CAP theorem. http:\/\/tinyurl.com\/3lqwebc.  N. Marz. How to beat the CAP theorem. http:\/\/tinyurl.com\/3lqwebc."},{"key":"e_1_2_1_26_1","unstructured":"Expression Trees in .NET. https:\/\/msdn.microsoft.com\/en-us\/library\/bb397951.aspx.  Expression Trees in .NET. https:\/\/msdn.microsoft.com\/en-us\/library\/bb397951.aspx."},{"key":"e_1_2_1_27_1","unstructured":"Microsoft Azure. https:\/\/azure.microsoft.com\/en-us\/.  Microsoft Azure. https:\/\/azure.microsoft.com\/en-us\/."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1989323.1989423"},{"key":"e_1_2_1_29_1","unstructured":"Oracle9i Database Performance Tuning Guide and Reference. Using Explain Plan. http:\/\/tinyurl.com\/hkh7zx6.  Oracle9i Database Performance Tuning Guide and Reference. Using Explain Plan. http:\/\/tinyurl.com\/hkh7zx6."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1559845.1559865"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.14778\/1687553.1687609"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2463676.2465288"},{"key":"e_1_2_1_33_1","volume-title":"NSDI","author":"Zaharia M.","year":"2012","unstructured":"M. Zaharia Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing . In NSDI , 2012 . M. Zaharia et al. Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. In NSDI, 2012."}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3007328.3007329","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T11:08:46Z","timestamp":1672225726000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3007328.3007329"}},"subtitle":["efficient, transferable, and rich analytics at scale"],"short-title":[],"issued":{"date-parts":[[2016,10]]},"references-count":33,"journal-issue":{"issue":"14","published-print":{"date-parts":[[2016,10]]}},"alternative-id":["10.14778\/3007328.3007329"],"URL":"https:\/\/doi.org\/10.14778\/3007328.3007329","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2016,10]]}}}