{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,11]],"date-time":"2026-01-11T04:43:41Z","timestamp":1768106621590,"version":"3.49.0"},"reference-count":56,"publisher":"Association for Computing Machinery (ACM)","issue":"4","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:p>The proliferation of modern data processing tools has given rise to open-source columnar data formats. These formats help organizations avoid repeated conversion of data to a new format for each application. However, these formats are read-only, and organizations must use a heavy-weight transformation process to load data from on-line transactional processing (OLTP) systems. As a result, DBMSs often fail to take advantage of full network bandwidth when transferring data. We aim to reduce or even eliminate this overhead by developing a storage architecture for in-memory database management systems (DBMSs) that is aware of the eventual usage of its data and emits columnar storage blocks in a universal open-source format. We introduce relaxations to common analytical data formats to efficiently update records and rely on a lightweight transformation process to convert blocks to a read-optimized layout when they are cold. We also describe how to access data from third-party analytical tools with minimal serialization overhead. We implemented our storage engine based on the Apache Arrow format and integrated it into the NoisePage DBMS to evaluate our work. Our experiments show that our approach achieves comparable performance with dedicated OLTP DBMSs while enabling orders-of-magnitude faster data exports to external data science and machine learning tools than existing methods.<\/jats:p>","DOI":"10.14778\/3436905.3436913","type":"journal-article","created":{"date-parts":[[2021,2,22]],"date-time":"2021-02-22T17:23:50Z","timestamp":1614014630000},"page":"534-546","source":"Crossref","is-referenced-by-count":15,"title":["Mainlining databases"],"prefix":"10.14778","volume":"14","author":[{"given":"Tianyu","family":"Li","sequence":"first","affiliation":[{"name":"Carnegie Mellon University"}]},{"given":"Matthew","family":"Butrovich","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University"}]},{"given":"Amadou","family":"Ngom","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University"}]},{"given":"Wan Shen","family":"Lim","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University"}]},{"given":"Wes","family":"McKinney","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University"}]},{"given":"Andrew","family":"Pavlo","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University"}]}],"member":"320","published-online":{"date-parts":[[2021,2,22]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"2013. olap4j: Open Java API for OLAP. http:\/\/www.olap4j.org.  2013. olap4j: Open Java API for OLAP. http:\/\/www.olap4j.org."},{"key":"e_1_2_1_2_1","unstructured":"2016. Guaranteed Atomic Operations on Intel Processors. https:\/\/www.intel.com\/content\/dam\/www\/public\/us\/en\/documents\/manuals\/64-ia-32-architectures-software-developer-system-programming-manual-325384.pdf.  2016. Guaranteed Atomic Operations on Intel Processors. https:\/\/www.intel.com\/content\/dam\/www\/public\/us\/en\/documents\/manuals\/64-ia-32-architectures-software-developer-system-programming-manual-325384.pdf."},{"key":"e_1_2_1_3_1","unstructured":"2018. Diving into the HANA DataFrame: Python Integration. https:\/\/blogs.sap.com\/2018\/12\/17\/diving-into-the-hana-dataframe-python-integration-part-1\/.  2018. Diving into the HANA DataFrame: Python Integration. https:\/\/blogs.sap.com\/2018\/12\/17\/diving-into-the-hana-dataframe-python-integration-part-1\/."},{"key":"e_1_2_1_4_1","unstructured":"2019. Apache Arrow. https:\/\/arrow.apache.org\/.  2019. Apache Arrow. https:\/\/arrow.apache.org\/."},{"key":"e_1_2_1_5_1","unstructured":"2019. Apache Arrow Source Code. https:\/\/github.com\/apache\/arrow.  2019. Apache Arrow Source Code. https:\/\/github.com\/apache\/arrow."},{"key":"e_1_2_1_6_1","unstructured":"2019. Apache Hive. https:\/\/hive.apache.org\/.  2019. Apache Hive. https:\/\/hive.apache.org\/."},{"key":"e_1_2_1_7_1","unstructured":"2019. Apache Impala. https:\/\/hive.apache.org\/.  2019. Apache Impala. https:\/\/hive.apache.org\/."},{"key":"e_1_2_1_8_1","unstructured":"2019. Apache Kudu. https:\/\/kudu.apache.org\/overview.html.  2019. Apache Kudu. https:\/\/kudu.apache.org\/overview.html."},{"key":"e_1_2_1_9_1","unstructured":"2019. Apache ORC. https:\/\/orc.apache.org\/.  2019. Apache ORC. https:\/\/orc.apache.org\/."},{"key":"e_1_2_1_10_1","unstructured":"2019. Apache Parquet. https:\/\/parquet.apache.org\/.  2019. Apache Parquet. https:\/\/parquet.apache.org\/."},{"key":"e_1_2_1_11_1","unstructured":"2019. Databricks Delta Lake. https:\/\/databricks.com\/blog\/2019\/04\/24\/open-sourcing-delta-lake.html.  2019. Databricks Delta Lake. https:\/\/databricks.com\/blog\/2019\/04\/24\/open-sourcing-delta-lake.html."},{"key":"e_1_2_1_12_1","unstructured":"2019. Dremio. https:\/\/docs.dremio.com\/.  2019. Dremio. https:\/\/docs.dremio.com\/."},{"key":"e_1_2_1_13_1","unstructured":"2019. IBM DB2 Pure Scale. https:\/\/www.ibm.com\/support\/knowledgecenter\/en\/SSEPGG10.5.0\/com.ibm.db2.luw.licensing.doc\/doc\/c0057442.html.  2019. IBM DB2 Pure Scale. https:\/\/www.ibm.com\/support\/knowledgecenter\/en\/SSEPGG10.5.0\/com.ibm.db2.luw.licensing.doc\/doc\/c0057442.html."},{"key":"e_1_2_1_14_1","unstructured":"2019. MemSQL SingleStore. https:\/\/www.memsql.com\/blog\/memsql-singlestore-then-there-was-one\/.  2019. MemSQL SingleStore. https:\/\/www.memsql.com\/blog\/memsql-singlestore-then-there-was-one\/."},{"key":"e_1_2_1_15_1","unstructured":"2019. Microsoft Analytics Platform System. https:\/\/www.microsoft.com\/en-us\/sql-server\/analytics-platform-system.  2019. Microsoft Analytics Platform System. https:\/\/www.microsoft.com\/en-us\/sql-server\/analytics-platform-system."},{"key":"e_1_2_1_16_1","unstructured":"2019. Microsoft SQL Server with SMB Direct. https:\/\/docs.microsoft.com\/en-us\/previous-versions\/windows\/it-pro\/windows-server-2012-R2-and-2012\/jj134210(v=ws.11).  2019. Microsoft SQL Server with SMB Direct. https:\/\/docs.microsoft.com\/en-us\/previous-versions\/windows\/it-pro\/windows-server-2012-R2-and-2012\/jj134210(v=ws.11)."},{"key":"e_1_2_1_17_1","unstructured":"2019. OmniSci GPU-Accelerated Analytics. https:\/\/www.omnisci.com\/.  2019. OmniSci GPU-Accelerated Analytics. https:\/\/www.omnisci.com\/."},{"key":"e_1_2_1_18_1","unstructured":"2019. Oracle Real Application Cluster. https:\/\/www.oracle.com\/technetwork\/server-storage\/networking\/documentation\/o12-020-1653901.pdf.  2019. Oracle Real Application Cluster. https:\/\/www.oracle.com\/technetwork\/server-storage\/networking\/documentation\/o12-020-1653901.pdf."},{"key":"e_1_2_1_19_1","unstructured":"2019. TensorFlow I\/O Apache Arrow Datasets. https:\/\/github.com\/tensorflow\/io\/tree\/master\/tensorflowio\/arrow.  2019. TensorFlow I\/O Apache Arrow Datasets. https:\/\/github.com\/tensorflow\/io\/tree\/master\/tensorflowio\/arrow."},{"key":"e_1_2_1_20_1","unstructured":"2019. Transaction Semantics in Apache Kudu. https:\/\/kudu.apache.org\/docs\/transactionsemantics.html.  2019. Transaction Semantics in Apache Kudu. https:\/\/kudu.apache.org\/docs\/transactionsemantics.html."},{"key":"e_1_2_1_21_1","unstructured":"2019. Using PyArrow from C++ and Cython Code. https:\/\/arrow.apache.org\/docs\/python\/extending.html.  2019. Using PyArrow from C++ and Cython Code. https:\/\/arrow.apache.org\/docs\/python\/extending.html."},{"key":"e_1_2_1_22_1","unstructured":"2020. Add a \"Flight SQL\" extension on top of FlightRPC. https:\/\/lists.apache.org\/thread.html\/rc4717b78f09bbf7a69347b6c126849e17323c491338fc73457cf7558%40%3Cdev.arrow.apache.org%3E.  2020. Add a \"Flight SQL\" extension on top of FlightRPC. https:\/\/lists.apache.org\/thread.html\/rc4717b78f09bbf7a69347b6c126849e17323c491338fc73457cf7558%40%3Cdev.arrow.apache.org%3E."},{"key":"e_1_2_1_23_1","unstructured":"2020. NoisePage. https:\/\/noise.page.  2020. NoisePage. https:\/\/noise.page."},{"key":"e_1_2_1_24_1","unstructured":"2020. Oracle TimesTen In-Memory Database. https:\/\/www.oracle.com\/database\/technologies\/related\/timesten.html.  2020. Oracle TimesTen In-Memory Database. https:\/\/www.oracle.com\/database\/technologies\/related\/timesten.html."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1376616.1376712"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-002-0074-9"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2610502"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2915231"},{"key":"e_1_2_1_29_1","unstructured":"Manos Athanassoulis Michael S. Kester Lukas M. Maas Radu Stoica Stratos Idreos Anastasia Ailamaki and Mark Callaghan. 2016. Designing Access Methods: The RUM Conjecture. In EDBT. 461--466.  Manos Athanassoulis Michael S. Kester Lukas M. Maas Radu Stoica Stratos Idreos Anastasia Ailamaki and Mark Callaghan. 2016. Designing Access Methods: The RUM Conjecture. In EDBT. 461--466."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/319996.319998"},{"key":"e_1_2_1_31_1","unstructured":"Peter Boncz Marcin Zukowski and Niels Nes. 2005. MonetDB\/X100: Hyper-pipelining query execution. In CIDR.  Peter Boncz Marcin Zukowski and Niels Nes. 2005. MonetDB\/X100: Hyper-pipelining query execution. In CIDR."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3196898"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.14778\/2556549.2556575"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/971697.602261"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.14778\/2732240.2732246"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.14778\/2350229.2350258"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.14778\/1921071.1921077"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.14778\/1453856.1453912"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2011.5767867"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.5555\/3275366.3284966"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2882925"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.14778\/2095686.2095689"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2903734"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.14778\/3151113.3151114"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/128765.128770"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2749436"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.14778\/3115404.3115408"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2915966"},{"key":"e_1_2_1_49_1","unstructured":"Mohammad Sadoghi Souvik Bhattacherjee Bishwaranjan Bhattacharjee and Mustafa Canim. 2018. L-Store: A Real-time OLTP and OLAP System. In Extending Database Technology. 540--551.  Mohammad Sadoghi Souvik Bhattacherjee Bishwaranjan Bhattacharjee and Mustafa Canim. 2018. L-Store: A Real-time OLTP and OLAP System. In Extending Database Technology. 540--551."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/2213836.2213946"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.5555\/1325851.1325981"},{"key":"e_1_2_1_52_1","unstructured":"The Transaction Processing Council. 2007. TPC-C Benchmark (Revision 5.9.0). http:\/\/www.tpc.org\/tpcc\/spec\/tpcccurrent.pdf.  The Transaction Processing Council. 2007. TPC-C Benchmark (Revision 5.9.0). http:\/\/www.tpc.org\/tpcc\/spec\/tpcccurrent.pdf."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/2517349.2522713"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3196895"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.14778\/3067421.3067427"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.14778\/2735508.2735511"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3436905.3436913","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T10:26:52Z","timestamp":1672223212000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3436905.3436913"}},"subtitle":["supporting fast transactional workloads on universal columnar data file formats"],"short-title":[],"issued":{"date-parts":[[2020,12]]},"references-count":56,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,12]]}},"alternative-id":["10.14778\/3436905.3436913"],"URL":"https:\/\/doi.org\/10.14778\/3436905.3436913","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2020,12]]}}}