{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T03:06:05Z","timestamp":1787022365973,"version":"build-2736575974"},"reference-count":85,"publisher":"Association for Computing Machinery (ACM)","issue":"13","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2019,9]]},"abstract":"<jats:p>Data-intensive analytical applications need to support both efficient reads and writes. However, what is usually a good data layout for an update-heavy workload, is not well-suited for a read-mostly one and vice versa. Modern analytical data systems rely on columnar layouts and employ delta stores to inject new data and updates.<\/jats:p>\n                  <jats:p>We show that for hybrid workloads we can achieve close to one order of magnitude better performance by tailoring the column layout design to the data and query workload. Our approach navigates the possible design space of the physical layout: it organizes each column's data by determining the number of partitions, their corresponding sizes and ranges, and the amount of buffer space and how it is allocated. We frame these design decisions as an optimization problem that, given workload knowledge and performance requirements, provides an optimal physical layout for the workload at hand. To evaluate this work, we build an in-memory storage engine, Casper, and we show that it outperforms state-of-the-art data layouts of analytical systems for hybrid workloads. Casper delivers up to 2.32x higher throughput for update-intensive workloads and up to 2.14x higher throughput for hybrid workloads. We further show how to make data layout decisions robust to workload variation by carefully selecting the input of the optimization.<\/jats:p>","DOI":"10.14778\/3358701.3358707","type":"journal-article","created":{"date-parts":[[2020,9,10]],"date-time":"2020-09-10T22:58:07Z","timestamp":1599778687000},"page":"2393-2407","source":"Crossref","is-referenced-by-count":31,"title":["Optimal column layout for hybrid workloads"],"prefix":"10.14778","volume":"12","author":[{"given":"Manos","family":"Athanassoulis","sequence":"first","affiliation":[{"name":"Boston University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kenneth S.","family":"B\u00f8gh","sequence":"additional","affiliation":[{"name":"Uber Technologies Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stratos","family":"Idreos","sequence":"additional","affiliation":[{"name":"Harvard University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2019,9]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.5555\/2602024"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1142473.1142548"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/B978-012088469-8.50097-8"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1007568.1007609"},{"key":"e_1_2_1_5_1","first-page":"169","volume-title":"Proceedings of the International Conference on Very Large Data Bases (VLDB)","author":"Ailamaki A.","year":"2001"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3064029"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2610502"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4757-3216-0_8"},{"key":"e_1_2_1_9_1","unstructured":"Apache. Parquet. https:\/\/parquet.apache.org\/.  Apache. Parquet. https:\/\/parquet.apache.org\/."},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of the Biennial Conference on Innovative Data Systems Research (CIDR)","author":"Appuswamy R.","year":"2017"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2915231"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1989323.1989414"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2699484"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2912569"},{"key":"e_1_2_1_15_1","first-page":"461","volume-title":"Proceedings of the International Conference on Extending Database Technology (EDBT)","author":"Athanassoulis M.","year":"2016"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2915964"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2015.7113372"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/SFCS.2000.892128"},{"issue":"4","key":"e_1_2_1_19_1","volume":"32","author":"Bender M. A.","year":"2007","journal-title":"An Adaptive Packed-Memory Array. ACM Transactions on Database Systems (TODS)"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/568271.223785"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1287\/opre.1090.0715"},{"key":"e_1_2_1_22_1","unstructured":"J. Bisschop. AIMMS - Optimization Modeling. AIMMS 2006.  J. Bisschop. AIMMS - Optimization Modeling. AIMMS 2006."},{"key":"e_1_2_1_23_1","first-page":"499","volume-title":"Proceedings of the International Conference on Very Large Data Bases (VLDB)","author":"Bruno N.","year":"2006"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2007.367928"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3196898"},{"key":"e_1_2_1_26_1","doi-asserted-by":"crossref","unstructured":"C. Chasseur and J. M. Patel. Design and Evaluation of Storage Organizations for Read-Optimized Main Memory Databases. PVLDB 6(13):1474--1485 2013.  C. Chasseur and J. M. Patel. Design and Evaluation of Storage Organizations for Read-Optimized Main Memory Databases. PVLDB 6(13):1474--1485 2013.","DOI":"10.14778\/2536258.2536260"},{"key":"e_1_2_1_27_1","first-page":"146","volume-title":"Proceedings of the International Conference on Very Large Data Bases (VLDB)","author":"Chaudhuri S.","year":"1997"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807128.1807152"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.14778\/1920841.1920853"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2903741"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3064054"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3276980"},{"issue":"1","key":"e_1_2_1_33_1","first-page":"28","volume":"35","author":"F\u00e4rber F.","year":"2012","journal-title":"The SAP HANA Database - An Architecture Overview. IEEE Data Engineering Bulletin"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/42201.42205"},{"key":"e_1_2_1_35_1","volume-title":"IBM Redbooks","author":"Francisco P.","year":"2011"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/1739041.1739087"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-07804-5_27"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807167.1807227"},{"issue":"1","key":"e_1_2_1_39_1","first-page":"40","volume":"35","author":"Idreos S.","year":"2012","journal-title":"IEEE Data Engineering Bulletin"},{"key":"e_1_2_1_40_1","volume-title":"Database Cracking. In Proceedings of the Biennial Conference on Innovative Data Systems Research (CIDR)","author":"Idreos S.","year":"2007"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1247480.1247527"},{"issue":"3","key":"e_1_2_1_42_1","first-page":"64","volume":"41","author":"Idreos S.","year":"2018","journal-title":"The Periodic Table of Data Structures. IEEE Data Engineering Bulletin"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3199671"},{"key":"e_1_2_1_44_1","first-page":"235","volume-title":"Deletes and Modifies. In Proceedings of the ACM Symposium on Principles of Database Systems (PODS)","author":"Johnson T.","year":"1989"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2011.5767867"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3196909"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2015.7113373"},{"key":"e_1_2_1_48_1","doi-asserted-by":"crossref","unstructured":"A. Lamb M. Fuller and R. Varadarajan. The Vertica Analytic Database: C-Store 7 Years Later. PVLDB 5(12):1790--1801 2012.  A. Lamb M. Fuller and R. Varadarajan. The Vertica Analytic Database: C-Store 7 Years Later. PVLDB 5(12):1790--1801 2012.","DOI":"10.14778\/2367502.2367518"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2882925"},{"issue":"12","key":"e_1_2_1_50_1","first-page":"1740","volume":"8","author":"Birka A.","year":"2015","journal-title":"Real-Time Analytical Processing with SQL Server. PVLDB"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/2463676.2463708"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDEW.2007.4401028"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3196908"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3035959"},{"key":"e_1_2_1_55_1","first-page":"476","volume-title":"Small Materialized Aggregates: A Light Weight Index Structure for Data Warehousing. In Proceedings of the International Conference on Very Large Data Bases (VLDB)","author":"Moerkotte G.","year":"1998"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2749454"},{"key":"e_1_2_1_57_1","unstructured":"MySQL. Online Reference for Storage Engine API. https:\/\/dev.mysql.com\/doc\/internals\/en\/custom-engine.html 2019.  MySQL. Online Reference for Storage Engine API. https:\/\/dev.mysql.com\/doc\/internals\/en\/custom-engine.html 2019."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2749436"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.14778\/3115404.3115415"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1007\/s002360050048"},{"key":"e_1_2_1_61_1","volume-title":"White Paper","year":"2018"},{"key":"e_1_2_1_62_1","first-page":"1771","volume-title":"Proceedings of the ACM SIGMOD International Conference on Management of Data","author":"\u00d6zcan F.","year":"2017"},{"issue":"10","key":"e_1_2_1_63_1","first-page":"610","volume":"4","author":"Pandis I.","year":"2011","journal-title":"PLP: Page Latch-free Shared-everything OLTP. PVLDB"},{"key":"e_1_2_1_64_1","volume-title":"Proceedings of the Biennial Conference on Innovative Data Systems Research (CIDR)","author":"Pavlo A.","year":"2017"},{"key":"e_1_2_1_65_1","first-page":"61","volume-title":"Parallel OLTP Systems. In Proceedings of the ACM SIGMOD International Conference on Management of Data","author":"Pavlo A.","year":"2012"},{"key":"e_1_2_1_66_1","unstructured":"M. Pezzini D. Feinberg N. Rayner and R. Edjlali. Hybrid Transaction\/Analytical Processing Will Foster Opportunities for Dramatic Business Innovation. https:\/\/www.gartner.com\/doc\/2657815\/ 2014.  M. Pezzini D. Feinberg N. Rayner and R. Edjlali. Hybrid Transaction\/Analytical Processing Will Foster Opportunities for Dramatic Business Innovation. https:\/\/www.gartner.com\/doc\/2657815\/ 2014."},{"issue":"11","key":"e_1_2_1_67_1","first-page":"1526","volume":"10","author":"Pilman M.","year":"2017","journal-title":"Fast Scans on Key-Value Stores. PVLDB"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/1559845.1559846"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/2452376.2452427"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1016\/B978-155860869-6\/50045-7"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-003-0093-1"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/3148.3161"},{"key":"e_1_2_1_73_1","doi-asserted-by":"crossref","first-page":"793","DOI":"10.1145\/1142473.1142592","volume-title":"Proceedings of the ACM SIGMOD International Conference on Management of Data","author":"Schnaitter K.","year":"2006"},{"key":"e_1_2_1_74_1","first-page":"16","volume-title":"Proceedings of the International Workshop on In Memory Data Management and Analytics (IMDM)","author":"Schwalb D.","year":"2014"},{"issue":"4","key":"e_1_2_1_75_1","first-page":"445","volume":"10","author":"Serafini M.","year":"2016","journal-title":"Clay: Fine-Grained Adaptive Partitioning for General Database Schemas. PVLDB"},{"key":"e_1_2_1_76_1","volume-title":"The MemSQL In-Memory Database System. In Proceedings of the International Workshop on In-Memory Data Management and Analytics (IMDM)","author":"Shamgunov N.","year":"2014"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.14778\/2536222.2536251"},{"key":"e_1_2_1_78_1","first-page":"553","volume-title":"Proceedings of the International Conference on Very Large Data Bases (VLDB)","author":"Stonebraker M.","year":"2005"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2610515"},{"issue":"4","key":"e_1_2_1_80_1","first-page":"421","volume":"10","author":"Sun L.","year":"2016","journal-title":"Skipping-oriented Partitioning for Columnar Layouts. PVLDB"},{"key":"e_1_2_1_81_1","unstructured":"TPC. Specification of TPC-H benchmark. http:\/\/www.tpc.org\/tpch\/ 2018.  TPC. Specification of TPC-H benchmark. http:\/\/www.tpc.org\/tpch\/ 2018."},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2011.5767830"},{"key":"e_1_2_1_83_1","first-page":"1087","volume-title":"Proceedings of the International Conference on Very Large Data Bases (VLDB)","author":"Zilio D. C.","year":"2004"},{"issue":"1","key":"e_1_2_1_84_1","first-page":"21","volume":"35","author":"Zukowski M.","year":"2012","journal-title":"Beyond Column Stores. IEEE Data Engineering Bulletin"},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2006.150"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3358701.3358707","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T04:54:41Z","timestamp":1672203281000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3358701.3358707"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,9]]},"references-count":85,"journal-issue":{"issue":"13","published-print":{"date-parts":[[2019,9]]}},"alternative-id":["10.14778\/3358701.3358707"],"URL":"https:\/\/doi.org\/10.14778\/3358701.3358707","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2019,9]]}}}