{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,15]],"date-time":"2026-09-15T06:43:40Z","timestamp":1789454620938,"version":"build-2803163510"},"reference-count":82,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2024,12,18]],"date-time":"2024-12-18T00:00:00Z","timestamp":1734480000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100006374","name":"European Research Council","doi-asserted-by":"publisher","award":["101041375"],"award-info":[{"award-number":["101041375"]}],"id":[{"id":"10.13039\/501100006374","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Manag. Data"],"published-print":{"date-parts":[[2024,12,18]]},"abstract":"<jats:p>This paper aims to bridge the gap between fast in-memory query engines and slow but robust engines that can utilize external storage. We find that current systems have to choose between fast in-memory operators and slower out-of-memory operators. We present a solution that leverages two independent but complementary techniques: First, we propose adaptive materialization, which can turn any hash-based in-memory operator into an out-of-memory operator without reducing in-memory performance. Second, we introduce self-regulating compression, which optimizes the throughput of spilling operators based on the current workload and available hardware. We evaluate these techniques using the prototype query engine Spilly, which matches the performance of state-of-the-art in-memory systems, but also efficiently executes large out-of-memory workloads by spilling to NVMe arrays.<\/jats:p>","DOI":"10.1145\/3698813","type":"journal-article","created":{"date-parts":[[2024,12,20]],"date-time":"2024-12-20T16:40:35Z","timestamp":1734712835000},"page":"1-27","source":"Crossref","is-referenced-by-count":12,"title":["High-Performance Query Processing with NVMe Arrays: Spilling without Killing Performance"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-6724-7530","authenticated-orcid":false,"given":"Maximilian","family":"Kuschewski","sequence":"first","affiliation":[{"name":"Technische Universit\u00e4t M\u00fcnchen, Munich, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1926-3551","authenticated-orcid":false,"given":"Jana","family":"Giceva","sequence":"additional","affiliation":[{"name":"Technische Universit\u00e4t M\u00fcnchen, Munich, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5787-142X","authenticated-orcid":false,"given":"Thomas","family":"Neumann","sequence":"additional","affiliation":[{"name":"Technische Universit\u00e4t M\u00fcnchen, Munich, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5676-8017","authenticated-orcid":false,"given":"Viktor","family":"Leis","sequence":"additional","affiliation":[{"name":"Technische Universit\u00e4t M\u00fcnchen, Munich, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,12,20]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"August 24 2022. DuckDB Out-of-Memory Hash Join. https:\/\/github.com\/duckdb\/duckdb\/pull\/4189"},{"key":"e_1_2_1_2_1","unstructured":"January 16 2023. DuckDB Benchmarking Guidelines. https:\/\/duckdb.org\/faq#i-benchmarked-duckdb-and-its-slowerthan-some-other-system"},{"key":"e_1_2_1_3_1","unstructured":"January 19 2023. AWS c7g instance family. https:\/\/aws.amazon.com\/de\/ec2\/instance-types\/c7g\/"},{"key":"e_1_2_1_4_1","doi-asserted-by":"crossref","unstructured":"January 19 2024. https:\/\/rocksdb.org\/","DOI":"10.21475\/ajcs.25.19.01"},{"key":"e_1_2_1_5_1","unstructured":"January 26 2023. Tableau Hyper API. https:\/\/tableau.github.io\/hyper-db\/docs\/"},{"key":"e_1_2_1_6_1","unstructured":"June 11 2024. https:\/\/geizhals.de\/micron-rdimm-32gb-mtc20f2085s1rc48ba1-a3017802.html"},{"key":"e_1_2_1_7_1","unstructured":"June 17 2024. AWS i3 instance family. https:\/\/aws.amazon.com\/ec2\/instance-types\/i3\/"},{"key":"e_1_2_1_8_1","unstructured":"June 17 2024. AWS i4i instance family. https:\/\/aws.amazon.com\/ec2\/instance-types\/i4i\/"},{"key":"e_1_2_1_9_1","unstructured":"March 26 2024. https:\/\/www.scylladb.com\/"},{"key":"e_1_2_1_10_1","unstructured":"November 1 2023. Elevate Performance with 5th Gen Intel Xeon Processors Featuring Intel Accelerator Engines. https:\/\/www.intel.com\/content\/dam\/www\/central-libraries\/us\/en\/documents\/2023--11\/5thgen-acceleratorengines-eguide.pdf"},{"key":"e_1_2_1_11_1","volume-title":"Intel 5th Generation Xeon Benchmarks. https:\/\/edc.intel.com\/content\/www\/us\/en\/products\/performance\/benchmarks\/5th-generation-intel-xeon-scalable-processors\/","author":"November","unstructured":"November 1, 2023. Intel 5th Generation Xeon Benchmarks. https:\/\/edc.intel.com\/content\/www\/us\/en\/products\/performance\/benchmarks\/5th-generation-intel-xeon-scalable-processors\/"},{"key":"e_1_2_1_12_1","unstructured":"November 16 2023. https:\/\/geizhals.de\/micron-5100-pro-960gb-mtfddak960tcb-1ar16abyy-a1562532.html"},{"key":"e_1_2_1_13_1","unstructured":"November 16 2023. Kioxia CM-7 Price History. https:\/\/www.idealo.de\/preisvergleich\/OffersOfProduct\/203225197_-cm7-r-3--84tb-kioxia.html"},{"key":"e_1_2_1_14_1","unstructured":"November 16 2023. Samsung PM1733 Price History. https:\/\/geizhals.de\/samsung-ssd-pm1733--3--84tb-mzwlj3t8hbls-00007-a2202065.html"},{"key":"e_1_2_1_15_1","unstructured":"November 16 2023. Samsung PM983 Price History. https:\/\/geizhals.de\/samsung-ssd-pm983--3--84tb-mzqlb3t8hals-00007-a1870387.html"},{"key":"e_1_2_1_16_1","volume-title":"Skew-Aware Join in Postgres. https:\/\/github.com\/postgres\/postgres\/blob\/611806cd\/src\/include\/executor\/hashjoin.h#L95","author":"October","unstructured":"October 26, 2023. Skew-Aware Join in Postgres. https:\/\/github.com\/postgres\/postgres\/blob\/611806cd\/src\/include\/executor\/hashjoin.h#L95"},{"key":"e_1_2_1_17_1","unstructured":"September 13 2023. Efficient IO with io_uring. https:\/\/kernel.dk\/io_uring.pdf"},{"key":"e_1_2_1_18_1","unstructured":"September 26 2023. DuckDB 0.9 Release Announcement. https:\/\/duckdb.org\/2023\/09\/26\/announcing-duckdb-090.html"},{"key":"e_1_2_1_19_1","unstructured":"Anastassia Ailamaki David J. DeWitt Mark D. Hill and Marios Skounakis. 2001. Weaving Relations for Cache Performance. In VLDB. Morgan Kaufmann 169--180."},{"key":"e_1_2_1_20_1","volume-title":"Wood","author":"Ailamaki Anastassia","year":"1999","unstructured":"Anastassia Ailamaki, David J. DeWitt, Mark D. Hill, and David A. Wood. 1999. DBMSs on a Modern Processor: Where Does Time Go?. In VLDB. Morgan Kaufmann, 266--277."},{"key":"e_1_2_1_21_1","doi-asserted-by":"crossref","unstructured":"Manos Athanassoulis Shimin Chen Anastasia Ailamaki Phillip B. Gibbons and Radu Stoica. 2011. MaSM: efficient online updates in data warehouses. In SIGMOD. ACM 865--876.","DOI":"10.1145\/1989323.1989414"},{"key":"e_1_2_1_22_1","doi-asserted-by":"crossref","unstructured":"Cagri Balkesen Jens Teubner Gustavo Alonso and M. Tamer \u00d6zsu. 2013. Main-memory hash joins on multi-core CPUs: Tuning to the underlying hardware. In ICDE. 362--373.","DOI":"10.1109\/ICDE.2013.6544839"},{"key":"e_1_2_1_23_1","doi-asserted-by":"crossref","unstructured":"Maximilian Bandle Jana Giceva and Thomas Neumann. 2021. To Partition or Not to Partition That is the Join Question in a Real System. In SIGMOD. 168--180.","DOI":"10.1145\/3448016.3452831"},{"key":"e_1_2_1_24_1","volume-title":"SIGMOD Conference. ACM, 37--48","author":"Blanas Spyros","unstructured":"Spyros Blanas, Yinan Li, and Jignesh M. Patel. 2011. Design and evaluation of main memory hash join algorithms for multi-core CPUs. In SIGMOD Conference. ACM, 37--48."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1409360.1409380"},{"key":"e_1_2_1_26_1","volume-title":"Kersten","author":"Boncz Peter A.","year":"1999","unstructured":"Peter A. Boncz, Stefan Manegold, and Martin L. Kersten. 1999. Database Architecture Optimized for the New Bottleneck: Memory Access. In VLDB. 54--65."},{"key":"e_1_2_1_27_1","volume-title":"Ross","author":"Cieslewicz John","year":"2007","unstructured":"John Cieslewicz and Kenneth A. Ross. 2007. Adaptive Aggregation on Chip Multiprocessors. In VLDB. 339--350."},{"key":"e_1_2_1_28_1","volume-title":"Wood","author":"DeWitt David J.","year":"1984","unstructured":"David J. DeWitt, Randy H. Katz, Frank Olken, Leonard D. Shapiro, Michael Stonebraker, and David A. Wood. 1984. Implementation Techniques for Main Memory Database Systems. In SIGMOD. 1--8."},{"key":"e_1_2_1_29_1","volume-title":"Patel","author":"Do Jaeyoung","year":"2009","unstructured":"Jaeyoung Do and Jignesh M. Patel. 2009. Join processing for flash SSDs: remembering past lessons. In DaMoN. 1--8."},{"key":"e_1_2_1_30_1","volume-title":"DeWitt","author":"Do Jaeyoung","year":"2013","unstructured":"Jaeyoung Do, Donghui Zhang, Jignesh M. Patel, and David J. DeWitt. 2013. Fast peak-to-peak behavior with SSD buffer pool. In ICDE. IEEE Computer Society, 1129--1140."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/1989323.1989442"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3568027"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.14778\/3611479.3611486"},{"key":"e_1_2_1_34_1","doi-asserted-by":"crossref","unstructured":"Philippe Flajolet \u00c9ric Fusy Olivier Gandouet and Fr\u00e9d\u00e9ric Meunier. 2007. HyperLogLog: the analysis of a near-optimal cardinality estimation algorithm. In AofA: Analysis of Algorithms.","DOI":"10.46298\/dmtcs.3545"},{"key":"e_1_2_1_35_1","doi-asserted-by":"crossref","unstructured":"Goetz Graefe. 2007. The five-minute rule twenty years later and how flash memory changes the rules. In DaMoN. 6.","DOI":"10.1145\/1363189.1363198"},{"key":"e_1_2_1_36_1","unstructured":"Goetz Graefe Ross Bunker and Shaun Cooper. 1998. Hash Joins and Hash Teams in Microsoft SQL Server. In VLDB. 86--97."},{"key":"e_1_2_1_37_1","first-page":"21","article-title":"Designing Database Operators for Flash-enabled Memory Hierarchies","volume":"33","author":"Graefe Goetz","year":"2010","unstructured":"Goetz Graefe, Stavros Harizopoulos, Harumi A. Kuno, Mehul A. Shah, Dimitris Tsirogiannis, and Janet L. Wiener. 2010. Designing Database Operators for Flash-enabled Memory Hierarchies. IEEE Data Eng. Bull. 33, 4 (2010), 21--27.","journal-title":"IEEE Data Eng. Bull."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1413254.1413261"},{"key":"e_1_2_1_39_1","unstructured":"Gabriel Haas Michael Haubenschild and Viktor Leis. 2020. Exploiting Directly-Attached NVMe Arrays in DBMS. In CIDR."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.14778\/3598581.3598584"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1376616.1376713"},{"key":"e_1_2_1_42_1","volume-title":"Early grouping gets the skew. Technical reports 2","author":"Helmer Sven","year":"2002","unstructured":"Sven Helmer, Thomas Neumann, and Guido Moerkotte. 2002. Early grouping gets the skew. Technical reports 2 (2002)."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.14778\/3632093.3632117"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.14778\/3547305.3547327"},{"key":"e_1_2_1_45_1","doi-asserted-by":"crossref","unstructured":"Alfons Kemper and Thomas Neumann. 2011. HyPer: A hybrid OLTP&OLAP main memory database system based on virtual memory snapshots. In ICDE. 195--206.","DOI":"10.1109\/ICDE.2011.5767867"},{"key":"e_1_2_1_46_1","first-page":"2209","article-title":"Everything You Always Wanted to Know About Compiled and Vectorized Queries But Were Afraid to Ask","volume":"11","author":"Kersten Timo","year":"2018","unstructured":"Timo Kersten, Viktor Leis, Alfons Kemper, Thomas Neumann, Andrew Pavlo, and Peter A. Boncz. 2018. Everything You Always Wanted to Know About Compiled and Vectorized Queries But Were Afraid to Ask. PVLDB 11, 13 (2018), 2209--2222.","journal-title":"PVLDB"},{"key":"e_1_2_1_47_1","doi-asserted-by":"crossref","unstructured":"Ioannis Koltsidas and Stratis Viglas. 2011. Data management over flash memory. In SIGMOD. 1209--1212.","DOI":"10.1145\/1989323.1989455"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE60146.2024.00288"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3589263"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.14778\/2095686.2095689"},{"key":"e_1_2_1_51_1","doi-asserted-by":"crossref","unstructured":"Viktor Leis Peter A. Boncz Alfons Kemper and Thomas Neumann. 2014. Morsel-driven parallelism: a NUMA-aware query evaluation framework for the many-core age. In SIGMOD. 743--754.","DOI":"10.1145\/2588555.2610507"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.14778\/2850583.2850594"},{"key":"e_1_2_1_53_1","doi-asserted-by":"crossref","unstructured":"Viktor Leis Michael Haubenschild Alfons Kemper and Thomas Neumann. 2018. LeanStore: In-Memory Data Management beyond Main Memory. In ICDE. 185--196.","DOI":"10.1109\/ICDE.2018.00026"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.14778\/2794367.2794375"},{"key":"e_1_2_1_55_1","first-page":"1","article-title":"The ART of practical synchronization","volume":"3","author":"Leis Viktor","year":"2016","unstructured":"Viktor Leis, Florian Scheibner, Alfons Kemper, and Thomas Neumann. 2016. The ART of practical synchronization. In DaMoN. 3:1--3:8.","journal-title":"DaMoN."},{"key":"e_1_2_1_56_1","volume-title":"Narasayya","author":"Li Feng","year":"2016","unstructured":"Feng Li, Sudipto Das, Manoj Syamala, and Vivek R. Narasayya. 2016. Accelerating Relational Databases by Leveraging Remote Memory and RDMA. In SIGMOD. ACM, 355--370."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.14778\/3007263.3007280"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2747644"},{"key":"e_1_2_1_59_1","unstructured":"Masaya Nakayama Masaru Kitsuregawa and Mikio Takagi. 1988. Hash-Partitioned Join Method Using Dynamic Destaging Strategy. In VLDB. 468--478."},{"key":"e_1_2_1_60_1","volume-title":"Freitag","author":"Neumann Thomas","year":"2020","unstructured":"Thomas Neumann and Michael J. Freitag. 2020. Umbra: A Disk-Based System with In-Memory Performance. In CIDR."},{"key":"e_1_2_1_61_1","first-page":"3","article-title":"Compiling Database Queries into Machine Code","volume":"37","author":"Neumann Thomas","year":"2014","unstructured":"Thomas Neumann and Viktor Leis. 2014. Compiling Database Queries into Machine Code. IEEE Data Eng. Bull. 37, 1 (2014), 3--11.","journal-title":"IEEE Data Eng. Bull."},{"key":"e_1_2_1_62_1","first-page":"1","article-title":"HPCache","volume":"7","author":"Nicholson Hamish","year":"2022","unstructured":"Hamish Nicholson, Periklis Chrysogelos, and Anastasia Ailamaki. 2022. HPCache: Memory-Efficient OLAP Through Proportional Caching. In DaMoN. 7:1--7:9.","journal-title":"Memory-Efficient OLAP Through Proportional Caching. In DaMoN."},{"key":"e_1_2_1_63_1","unstructured":"Hamish Nicholson Aunn Raza Periklis Chrysogelos and Anastasia Ailamaki. 2023. HetCache: Synergising NVMe Storage and GPU acceleration for Memory-Efficient Analytics. In CIDR."},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2018.00047"},{"key":"e_1_2_1_65_1","unstructured":"Tarikul Islam Papon and Manos Athanassoulis. 2021. The Need for a New I\/O Model. In CIDR. www.cidrdb.org."},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.14778\/3199517.3199518"},{"key":"e_1_2_1_67_1","volume-title":"Ross","author":"Polychroniou Orestis","year":"2014","unstructured":"Orestis Polychroniou and Kenneth A. Ross. 2014. A comprehensive study of main-memory partitioning and its application to large-scale comparison- and radix-sort. In SIGMOD. 755--766."},{"key":"e_1_2_1_68_1","first-page":"1","article-title":"Fair Benchmarking Considered Difficult","volume":"2","author":"Raasveldt Mark","year":"2018","unstructured":"Mark Raasveldt, Pedro Holanda, Tim Gubner, and Hannes M\u00fchleisen. 2018. Fair Benchmarking Considered Difficult: Common Pitfalls In Database Performance Testing. In DBTest. 2:1--2:6.","journal-title":"Common Pitfalls In Database Performance Testing. In DBTest."},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3320212"},{"key":"e_1_2_1_70_1","doi-asserted-by":"crossref","unstructured":"Bogdan Raducanu Peter A. Boncz and Marcin Zukowski. 2013. Micro adaptivity in Vectorwise. In SIGMOD. 1231--1242.","DOI":"10.1145\/2463676.2465292"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.14778\/2536222.2536233"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.14778\/2536222.2536226"},{"key":"e_1_2_1_73_1","volume-title":"An Experimental Comparison of Thirteen Relational Equi-Joins in Main Memory. In SIGMOD Conference. ACM","author":"Schuh Stefan","year":"2016","unstructured":"Stefan Schuh, Xiao Chen, and Jens Dittrich. 2016. An Experimental Comparison of Thirteen Relational Equi-Joins in Main Memory. In SIGMOD Conference. ACM, 1961--1976."},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.14778\/2777598.2777602"},{"key":"e_1_2_1_75_1","unstructured":"Michael Stonebraker Samuel Madden Daniel J. Abadi Stavros Harizopoulos Nabil Hachem and Pat Helland. 2007. The End of an Architectural Era (It's Time for a Complete Rewrite). In VLDB. ACM 1150--1160."},{"key":"e_1_2_1_76_1","volume-title":"Wiener, and Goetz Graefe","author":"Tsirogiannis Dimitris","year":"2009","unstructured":"Dimitris Tsirogiannis, Stavros Harizopoulos, Mehul A. Shah, Janet L. Wiener, and Goetz Graefe. 2009. Query processing techniques for solid state drives. In SIGMOD. 59--72."},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.14778\/3583140.3583156"},{"key":"e_1_2_1_78_1","unstructured":"Leonard von Merzljak Philipp Fent Thomas Neumann and Jana Giceva. 2022. What Are You Waiting For? Use Coroutines for Asynchronous I\/O to Hide I\/O Latencies and Maximize the Read Bandwidth!. In ADMS."},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/362084.362137"},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.14778\/3503585.3503587"},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1145\/3626739"},{"key":"e_1_2_1_82_1","volume-title":"Boncz","author":"Zukowski Marcin","year":"2012","unstructured":"Marcin Zukowski, Mark van de Wiel, and Peter A. Boncz. 2012. Vectorwise: A Vectorized Analytical DBMS. In ICDE. 1349--1350."}],"container-title":["Proceedings of the ACM on Management of Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3698813","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3698813","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,31]],"date-time":"2026-03-31T17:45:44Z","timestamp":1774979144000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3698813"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,18]]},"references-count":82,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2024,12,18]]}},"alternative-id":["10.1145\/3698813"],"URL":"https:\/\/doi.org\/10.1145\/3698813","relation":{},"ISSN":["2836-6573"],"issn-type":[{"value":"2836-6573","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,18]]}}}