{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T06:27:02Z","timestamp":1778048822084,"version":"3.51.4"},"reference-count":82,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,3,12]],"date-time":"2024-03-12T00:00:00Z","timestamp":1710201600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"USA Department of Energy project","award":["DE-SC0020200."],"award-info":[{"award-number":["DE-SC0020200."]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Manag. Data"],"published-print":{"date-parts":[[2024,3,12]]},"abstract":"<jats:p>We present Limousine, a self-designing key-value storage engine, that can automatically morph to the near-optimal storage engine architecture shape given a workload, a cloud budget, and target performance. At its core, Limousine identifies the fundamental design principles of storage engines as combinations of learned and classical data structures that collaborate through algorithms for data storage and access. By unifying these principles over diverse hardware and three major cloud providers (AWS, GCP, and Azure), Limousine creates a massive design space of quindecillion (1048) storage engine designs the vast majority of which do not exist in literature or industry. Limousine contains a distribution-aware IO model to accurately evaluate any candidate design. Using these models, Limousine searches within the exhaustive design space to construct a navigable continuum of designs connected along a Pareto frontier of cloud cost and performance. If storage engines contain learned components, Limousine also introduces efficient lazy write algorithms to optimize the holistic read-write performance. Once the near-optimal design is decided for the given context, Limousine automatically materializes the corresponding design in Rust code. Using the YCSB benchmark, we demonstrate that storage engines automatically designed and generated by Limousine scale better by up to 3 orders of magnitude when compared with state-of-the-art industry-leading engines such as RocksDB, WiredTiger, FASTER, and Cosine, over diverse workloads, data sets, and cloud budgets.<\/jats:p>","DOI":"10.1145\/3639302","type":"journal-article","created":{"date-parts":[[2024,3,26]],"date-time":"2024-03-26T18:51:32Z","timestamp":1711479092000},"page":"1-28","source":"Crossref","is-referenced-by-count":10,"title":["Limousine: Blending Learned and Classical Indexes to Self-Design Larger-than-Memory Cloud Storage Engines"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3735-5721","authenticated-orcid":false,"given":"Subarna","family":"Chatterjee","sequence":"first","affiliation":[{"name":"Harvard University, Cambridge, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-9602-6927","authenticated-orcid":false,"given":"Mark F.","family":"Pekala","sequence":"additional","affiliation":[{"name":"Harvard University, Cambridge, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-2965-512X","authenticated-orcid":false,"given":"Lev","family":"Kruglyak","sequence":"additional","affiliation":[{"name":"Harvard University, Cambridge, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8271-9187","authenticated-orcid":false,"given":"Stratos","family":"Idreos","sequence":"additional","affiliation":[{"name":"Harvard University, Cambridge, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,3,26]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"2019. AWS Calculator. https:\/\/calculator.s3.amazonaws.com\/index.html."},{"key":"e_1_2_1_2_1","unstructured":"2019. Azure Calculator. https:\/\/azure.microsoft.com\/en-us\/pricing\/."},{"key":"e_1_2_1_3_1","unstructured":"2019. GCP Calculator. https:\/\/cloud.google.com\/products\/calculator\/."},{"key":"e_1_2_1_4_1","unstructured":"2019. Microsoft Azure. https:\/\/azure.microsoft.com\/en-us\/pricing\/details\/virtual-machines\/linux\/."},{"key":"e_1_2_1_5_1","unstructured":"2020. General purpose Azure VMs. https:\/\/docs.microsoft.com\/en-us\/azure\/virtual-machines\/sizes-general?toc=\/azure\/virtual-machines\/linux\/toc.json&bc=\/azure\/virtual-machines\/linux\/breadcrumb\/toc.json."},{"key":"e_1_2_1_6_1","unstructured":"2024. Amazon OpenStreetMap on AWS. https:\/\/www.nyc.gov\/site\/tlc\/about\/tlc-trip-record-data.page."},{"key":"e_1_2_1_7_1","unstructured":"2024. Apache Cassandra. http:\/\/cassandra.apache.org."},{"key":"e_1_2_1_8_1","unstructured":"2024. CouchDB. http:\/\/couchdb.apache.org\/."},{"key":"e_1_2_1_9_1","unstructured":"2024. Facebook RocksDB. https:\/\/github.com\/facebook\/rocksdb."},{"key":"e_1_2_1_10_1","unstructured":"2024. MongoDB. http:\/\/www.mongodb.com\/."},{"key":"e_1_2_1_11_1","unstructured":"2024. WiredTiger. https:\/\/github.com\/wiredtiger\/wiredtiger."},{"key":"e_1_2_1_12_1","volume-title":"Lyric Doshi, Tim Kraska, Xiaozhou Li, Andy Ly, and Christopher Olston.","author":"Abu-Libdeh Hussam","year":"2020","unstructured":"Hussam Abu-Libdeh, Deniz Altinbuken, Alex Beutel, Ed H. Chi, Lyric Doshi, Tim Kraska, Xiaozhou Li, Andy Ly, and Christopher Olston. 2020. Learned Indexes for a Google-scale Disk-based Database. NeurIPS abs\/2012.12501 (2020)."},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the 26th International Conference on Very Large Data Bases (VLDB '00)","author":"Agrawal Sanjay","unstructured":"Sanjay Agrawal, Surajit Chaudhuri, and Vivek R. Narasayya. 2000. Automated Selection of Materialized Views and Indexes in SQL Databases. In Proceedings of the 26th International Conference on Very Large Data Bases (VLDB '00). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 496--505."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1007568.1007609"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3397536.3426358"},{"key":"e_1_2_1_16_1","doi-asserted-by":"crossref","unstructured":"G. Amdahl. 1967. Validity of the Single-Processor Approach to Achieving Large-Scale Computing Capabilities. In AFIPS spring joint computer conference.","DOI":"10.1145\/1465482.1465560"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2463676.2465296"},{"key":"e_1_2_1_18_1","volume-title":"Designing Access Methods: The RUM Conjecture. In International Conference on Extending Database Technology.","author":"Athanassoulis Manos","unstructured":"Manos Athanassoulis, Michael S. Kester, Lukas M. Maas, Radu Stoica, Stratos Idreos, Anastasia Ailamaki, and Mark D. Callaghan. 2016. Designing Access Methods: The RUM Conjecture. In International Conference on Extending Database Technology."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.5555\/644108.644201"},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the Biennial Conference on Innovative Data Systems Research (CIDR). http:\/\/cidrdb.org\/cidr2013\/Papers\/CIDR13{_}Paper46","author":"Cao Zhao","year":"2013","unstructured":"Zhao Cao, Shimin Chen, Feifei Li, Min Wang, and Xiaoyang Sean Wang. 2013. LogKV: Exploiting Key-Value Stores for Log Processing. In Proceedings of the Biennial Conference on Innovative Data Systems Research (CIDR). http:\/\/cidrdb.org\/cidr2013\/Papers\/CIDR13{_}Paper46.pdf"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3196898"},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the VLDB Endowment.","author":"Chatterjee Subarna","year":"2022","unstructured":"Subarna Chatterjee, Meena Jagadeesan, Wilson Qin, and Stratos Idreos. 2022. Cosine: A Cloud-Cost Optimized Self-Designing Key-Value Storage Engine. In Proceedings of the VLDB Endowment."},{"key":"e_1_2_1_23_1","unstructured":"Subarna Chatterjee Mark Pekala Lev Kruglyak and Stratos Idreos. 2024. Technical Report for Limousine."},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the 33rd International Conference on Very Large Data Bases","author":"Chaudhuri Surajit","year":"2007","unstructured":"Surajit Chaudhuri and Vivek Narasayya. 2007. Self-Tuning Database Systems: A Decade of Progress. In Proceedings of the 33rd International Conference on Very Large Data Bases (Vienna, Austria) (VLDB '07). VLDB Endowment, 3--14."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/356770.356776"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/356770.356776"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807128.1807152"},{"key":"e_1_2_1_28_1","unstructured":"G. Cudahy. 2022. The Key To Intelligent Connectivity In A World Awash With IoT Data? Making Decisions At The Edge. https:\/\/www.forbes.com\/sites\/forbestechcouncil\/2022\/03\/04\/the-key-to-intelligent-connectivity-in-a-world-awash-with-iot-data-making-decisions-at-the-edge\/'sh=7521b0794b13"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.14778\/1920841.1920853"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1016\/B978--155860869--6\/50095-0"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3064054"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3196927"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3319903"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1294261.1294281"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3318464.3389711"},{"key":"e_1_2_1_36_1","volume-title":"Set Theory: An Introduction to Large Cardinals. Studies in Logic & the Foundations of Mathematics","author":"Drake F. R.","year":"1974","unstructured":"F. R. Drake. 1974. Set Theory: An Introduction to Large Cardinals. Studies in Logic & the Foundations of Mathematics, Elsevier Science Ltd. 76 (1974)."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.14778\/1687627.1687767"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2018.00094"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.5555\/3524938.3525231"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.14778\/3389133.3389135"},{"key":"e_1_2_1_41_1","volume-title":"SQLGraph: An Efficient Relational-Based Property Graph Store. In In Proceedings of the International Conference on Management of Data, SIGMOD.","author":"Fokoue W.","unstructured":"W. Fokoue, A. Fokoue, K. Srinivas, A. Kementsietsidis, G. Hu, and G. Xie. 2015. SQLGraph: An Efficient Relational-Based Property Graph Store. In In Proceedings of the International Conference on Management of Data, SIGMOD."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3319860"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1561\/1900000028"},{"key":"e_1_2_1_44_1","volume-title":"Computer Architecture: A Quantitative Approach. Morgan Kauffman.","author":"Hennessy J. L.","year":"2003","unstructured":"J. L. Hennessy and D. A. Patterson. 2003. Computer Architecture: A Quantitative Approach. Morgan Kauffman."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2008.209"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3314041"},{"key":"e_1_2_1_47_1","doi-asserted-by":"crossref","unstructured":"S. Idreos and M. Callaghan. 2020. Key-Value Storage Engines. In ACM SIGMOD Tutorial.","DOI":"10.1145\/3318464.3383133"},{"key":"e_1_2_1_48_1","volume-title":"Biennial Conference on Innovative Data Systems Research (CIDR).","author":"Idreos Stratos","year":"2019","unstructured":"Stratos Idreos, Niv Dayan, Wilson Qin, Mali Akmanalp, Sophie Hilgard, Andrew Ross, James Lennon, Varun Jain, Harshita Gupta, David Li, and Zichen Zhu. 2019. Design Continuums and the Path Toward Self-Designing Key-Value Stores that Know and Learn. In Biennial Conference on Innovative Data Systems Research (CIDR)."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3314034"},{"key":"e_1_2_1_50_1","first-page":"64","article-title":"The Periodic Table of Data Structures","volume":"41","author":"Idreos Stratos","year":"2018","unstructured":"Stratos Idreos, Konstantinos Zoumpatianos, Manos Athanassoulis, Niv Dayan, Brian Hentschel, Michael S. Kester, Demi Guo, Lukas M. Maas, Wilson Qin, Abdul Wasay, and Yiyou Sun. 2018. The Periodic Table of Data Structures. IEEE Data Eng. Bull. 41 (2018), 64--75.","journal-title":"IEEE Data Eng. Bull."},{"key":"e_1_2_1_51_1","first-page":"47","article-title":"Learning Data Structure Alchemy","volume":"42","author":"Idreos Stratos","year":"2019","unstructured":"Stratos Idreos, Kostas Zoumpatianos, Subarna Chatterjee, Wilson Qin, Abdul Wasay, Brian Hentschel, Michael S. Kester, Niv Dayan, Demi Guo, Minseo Kang, and Yiyou Sun. 2019. Learning Data Structure Alchemy. IEEE Data Eng. Bull. 42 (2019), 47--58.","journal-title":"IEEE Data Eng. Bull."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3199671"},{"key":"e_1_2_1_53_1","unstructured":"M. R. Jain. 2019. Why we choose Badger over RocksDB in Dgraph. https:\/\/blog.dgraph.io\/post\/badger-over-rocksdb-in-dgraph\/."},{"key":"e_1_2_1_54_1","volume-title":"Proceedings of the 13th USENIX Conference on File and Storage Technologies","author":"Jannen William","unstructured":"William Jannen, Jun Yuan, Yang Zhan, Amogh Akshintala, John Esmet, Yizheng Jiao, Ankur Mittal, Prashant Pandey, Phaneendra Reddy, Leif Walsh, Michael Bender, Martin Farach-Colton, Rob Johnson, Bradley C. Kuszmaul, and Donald E. Porter. 2015. BetrFS: A Right-Optimized Write-Optimized File System. In Proceedings of the 13th USENIX Conference on File and Storage Technologies (Santa Clara, CA) (FAST'15). USENIX Association, USA, 301--315."},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3401071.3401659"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3196909"},{"key":"e_1_2_1_57_1","unstructured":"Eva Kwan Sam Lightstone Berni Schiefer Adam Storm and Leanne Wu. 2003. Automatic Database Configuration for DB2 Universal Database: Compressing Years of Performance Expertise into Seconds of Execution. 620--629."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.14778\/3489496.3489512"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.14778\/1920841.1920990"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3196908"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3448016.3457276"},{"key":"e_1_2_1_62_1","unstructured":"P. Malkowski. 2018. MyRocks Disk Full Edge Case. https:\/\/www.percona.com\/blog\/2018\/09\/20\/myrocks-disk-full-edge-case\/."},{"key":"e_1_2_1_63_1","unstructured":"W. Oledzki. 2013. memcached is a weird creature. http:\/\/hoborglabs.com\/en\/blog\/2013\/memcached-php."},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.14778\/3476311.3476411"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/2213836.2213844"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/335191.335449"},{"key":"e_1_2_1_67_1","volume-title":"Worldwide IDC Global DataSphere Forecast","author":"Rydning J.","year":"2022","unstructured":"J. Rydning. 2022. Worldwide IDC Global DataSphere Forecast, 2022--2026: Enterprise Organizations Driving Most of the Data Growth. https:\/\/www.idc.com\/getdoc.jsp?containerId=US49018922"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/3514221.3522563"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3514221.3522563"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/3514221.3520169"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/3318464.3389757"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.14778\/3476249.3476274"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/3639307"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.14778\/3594512.3594528"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3064029"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/3409963.3410496"},{"key":"e_1_2_1_77_1","doi-asserted-by":"crossref","unstructured":"Z. Wei G. Pierre and C. H. Chi. 2011. CloudTPS: Scalable Transactions for Web Applications in the Cloud. IEEE Transactions on Services Computing (2011) 525--539.","DOI":"10.1109\/TSC.2011.18"},{"key":"e_1_2_1_78_1","unstructured":"C. Weinschenk. 2018. Deeper Dive-Will growth in data traffic ever slow down? https:\/\/www.fiercewireless.com\/wireless\/special-report-will-growth-data-traffic-ever-slow-down"},{"key":"e_1_2_1_79_1","unstructured":"J. Wise. 2022. How Much Data Is Created Every Day In 2022? http:\/\/tinyurl.com\/dataCreatedEveryday"},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2018.00092"},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.14778\/3342263.3342630"},{"key":"e_1_2_1_82_1","volume-title":"Proceedings of the 26th International Conference on Extending Database Technology (EDBT) (EDBT '23)","author":"Heinis Thomas Yang Guang Hadian Ali","year":"2023","unstructured":"Hadian Ali Heinis Thomas Yang Guang, Liang Liang. 2023. FLIRT: A Fast Learned Index for Rolling Time frames. In Proceedings of the 26th International Conference on Extending Database Technology (EDBT) (EDBT '23). Association for Computing Machinery."}],"container-title":["Proceedings of the ACM on Management of Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3639302","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3639302","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,21]],"date-time":"2025-08-21T15:12:32Z","timestamp":1755789152000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3639302"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,12]]},"references-count":82,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,3,12]]}},"alternative-id":["10.1145\/3639302"],"URL":"https:\/\/doi.org\/10.1145\/3639302","relation":{},"ISSN":["2836-6573"],"issn-type":[{"value":"2836-6573","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,12]]}}}