{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,28]],"date-time":"2026-03-28T20:21:31Z","timestamp":1774729291893,"version":"3.50.1"},"reference-count":51,"publisher":"Association for Computing Machinery (ACM)","issue":"12","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2025,8]]},"abstract":"<jats:p>This retrospective revisits our 2015 PVLDB paper How Good Are Query Optimizers, Really?, which challenged the prevailing notion that query optimization was a solved problem. By designing the Join Order Benchmark (JOB) and conducting a series of systematic experiments, we empirically disentangled the contributions of plan enumeration, cost modeling, and cardinality estimation. Our findings showed that cardinality estimation errors are widespread and often the dominant factor behind poor query plans, while cost models and enumeration strategies matter comparatively less. The benchmark and methodology helped refocus the community's attention on cardinality estimation and led to a resurgence of research in this area, including learned and AI-based approaches. We reflect on the role of experiments and benchmarking in database research, survey developments in query optimization over the past decade, and discuss open challenges around robustness, adaptive execution, and realistic workloads.<\/jats:p>","DOI":"10.14778\/3750601.3760521","type":"journal-article","created":{"date-parts":[[2025,9,16]],"date-time":"2025-09-16T13:38:05Z","timestamp":1758029885000},"page":"5531-5536","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Still Asking: How Good Are Query Optimizers, Really?"],"prefix":"10.14778","volume":"18","author":[{"given":"Viktor","family":"Leis","sequence":"first","affiliation":[{"name":"Technische Universit\u00e4t M\u00fcnchen, Germany"}]},{"given":"Andrey","family":"Gubichev","sequence":"additional","affiliation":[{"name":"Databricks, USA"}]},{"given":"Atanas","family":"Mirchev","sequence":"additional","affiliation":[{"name":"Volkswagen Group, Germany"}]},{"given":"Peter","family":"Boncz","sequence":"additional","affiliation":[{"name":"CWI, Netherlands"}]},{"given":"Alfons","family":"Kemper","sequence":"additional","affiliation":[{"name":"Technische Universit\u00e4t M\u00fcnchen, Germany"}]},{"given":"Thomas","family":"Neumann","sequence":"additional","affiliation":[{"name":"Technische Universit\u00e4t M\u00fcnchen, Germany"}]}],"member":"320","published-online":{"date-parts":[[2025,9,16]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"34th International Conference on Very Large Data: Calls. https:\/\/www.cs.auckland.ac.nz\/research\/conferences\/vldb_site\/calls.html#125","unstructured":"2008. 34th International Conference on Very Large Data: Calls. https:\/\/www.cs.auckland.ac.nz\/research\/conferences\/vldb_site\/calls.html#125"},{"key":"e_1_2_1_2_1","volume-title":"Instance-Optimal Acyclic Join Processing Without Regret: Engineering the Yannakakis Algorithm in Column Stores. CoRR abs\/2411.04042","author":"Bekkers Liese","year":"2024","unstructured":"Liese Bekkers, Frank Neven, Stijn Vansummeren, and Yisu Remy Wang. 2024. Instance-Optimal Acyclic Join Processing Without Regret: Engineering the Yannakakis Algorithm in Column Stores. CoRR abs\/2411.04042 (2024)."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.14778\/3681954.3681995"},{"key":"e_1_2_1_4_1","doi-asserted-by":"crossref","unstructured":"Surajit Chaudhuri. 1998. An Overview of Query Optimization in Relational Systems. In PODS. 34\u201343.","DOI":"10.1145\/275487.275492"},{"key":"e_1_2_1_5_1","first-page":"1","volume-title":"Proc. ACM Manag. Data 1","author":"Deeds Kyle B.","year":"2023","unstructured":"Kyle B. Deeds, Dan Suciu, and Magdalena Balazinska. 2023. SafeBound: A Practical System for Generating Cardinality Bounds. Proc. ACM Manag. Data 1, 1 (2023), 53:1\u201353:26."},{"key":"e_1_2_1_6_1","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1145\/3567464","article-title":"DIAMETRICS: benchmarking query engines at scale","volume":"65","author":"Deep Shaleen","year":"2022","unstructured":"Shaleen Deep, Anja Gruenheid, Kruthi Nagaraj, Hiro Naito, Jeffrey F. Naughton, and Stratis Viglas. 2022. DIAMETRICS: benchmarking query engines at scale. Commun. ACM 65, 12 (2022), 105\u2013112.","journal-title":"Commun. ACM"},{"key":"e_1_2_1_7_1","first-page":"3","article-title":"Extensible Query Optimizers in Practice","volume":"14","author":"Ding Bailu","year":"2024","unstructured":"Bailu Ding, Vivek R. Narasayya, and Surajit Chaudhuri. 2024. Extensible Query Optimizers in Practice. Found. Trends Databases 14, 3\u20134 (2024), 186\u2013402.","journal-title":"Found. Trends Databases"},{"key":"e_1_2_1_8_1","volume-title":"Proc. SIGMOD Conference. ACM, 619\u2013630","author":"Erling Orri","unstructured":"Orri Erling, Alex Averbuch, Josep Llu\u00eds Larriba-Pey, Hassan Chafi, Andrey Gubichev, Arnau Prat-P\u00e9rez, Minh-Duc Pham, and Peter A. Boncz. 2015. The LDBC Social Network Benchmark: Interactive Workload. In Proc. SIGMOD Conference. ACM, 619\u2013630."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.14778\/3407790.3407797"},{"key":"e_1_2_1_10_1","volume-title":"Boncz","author":"Gubichev Andrey","year":"2014","unstructured":"Andrey Gubichev and Peter A. Boncz. 2014. Parameter Curation for Benchmark Queries. In Proc. TPCTC (LNCS, Vol. 8904). Springer, 113\u2013129."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.14778\/3503585.3503586"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.14778\/3384345.3384349"},{"key":"e_1_2_1_13_1","unstructured":"Matteo Interlandi. 2024. Running a Query Optimizer Advisor in Production: What we Learned (and What the Model didn't). https:\/\/hpts.ws\/papers\/2024\/2024_session9_interlandi.pdf"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.14778\/3648160.3648175"},{"key":"e_1_2_1_15_1","volume-title":"Seo, Wook-Shin Han, Kangwoo Choi, and Jaehyok Chong. 2022. Learned Cardinality Estimation: An In-depth Study. In SIGMOD Conference. 1214\u20131227","author":"Kim Kyoungmin","unstructured":"Kyoungmin Kim, Jisung Jung, In Seo, Wook-Shin Han, Kangwoo Choi, and Jaehyok Chong. 2022. Learned Cardinality Estimation: An In-depth Study. In SIGMOD Conference. 1214\u20131227."},{"key":"e_1_2_1_16_1","volume-title":"Learned Cardinalities: Estimating Correlated Joins with Deep Learning. In CIDR.","author":"Kipf Andreas","year":"2019","unstructured":"Andreas Kipf, Thomas Kipf, Bernhard Radke, Viktor Leis, Peter A. Boncz, and Alfons Kemper. 2019. Learned Cardinalities: Estimating Correlated Joins with Deep Learning. In CIDR."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.14778\/3611479.3611494"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.14778\/2850583.2850594"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-017-0480-7"},{"key":"e_1_2_1_20_1","unstructured":"Guy Lohman. 2014. Is Query Optimization a \"Solved\" Problem? https:\/\/wp.sigmod.org\/?p=1075"},{"key":"e_1_2_1_21_1","volume-title":"Bao: Making Learned Query Optimization Practical. In SIGMOD Conference. 1275\u20131288","author":"Marcus Ryan","year":"2021","unstructured":"Ryan Marcus, Parimarjan Negi, Hongzi Mao, Nesime Tatbul, Mohammad Alizadeh, and Tim Kraska. 2021. Bao: Making Learned Query Optimization Practical. In SIGMOD Conference. 1275\u20131288."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.14778\/3342263.3342644"},{"key":"e_1_2_1_23_1","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1007\/s00778-006-0030-1","article-title":"Consistent selectivity estimation via maximum entropy","volume":"16","author":"Markl Volker","year":"2007","unstructured":"Volker Markl, Peter J. Haas, Marcel Kutsch, Nimrod Megiddo, Utkarsh Srivastava, and Tam Minh Tran. 2007. Consistent selectivity estimation via maximum entropy. VLDB J. 16, 1 (2007), 55\u201376.","journal-title":"VLDB J."},{"key":"e_1_2_1_24_1","volume-title":"Peter J. Haas, and Utkarsh Srivastava.","author":"Markl Volker","year":"2005","unstructured":"Volker Markl, Nimrod Megiddo, Marcel Kutsch, Tam Minh Tran, Peter J. Haas, and Utkarsh Srivastava. 2005. Consistently Estimating the Selectivity of Conjuncts of Predicates. In VLDB. 373\u2013384."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.14778\/3342263.3342643"},{"key":"e_1_2_1_26_1","unstructured":"Guido Moerkotte. 2023. Building Query Compilers (Draft \/ Under Construction)."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.14778\/3476249.3476259"},{"key":"e_1_2_1_28_1","doi-asserted-by":"crossref","unstructured":"Hung Q. Ngo. 2018. Worst-Case Optimal Join Algorithms: Techniques Results and Open Problems. In PODS Jan Van den Bussche and Marcelo Arenas (Eds.). ACM 111\u2013124.","DOI":"10.1145\/3196959.3196990"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180143"},{"key":"e_1_2_1_30_1","doi-asserted-by":"crossref","first-page":"1022","DOI":"10.14778\/3115404.3115408","article-title":"Don't Hold My Data Hostage - A Case For Client Protocol Redesign","volume":"10","author":"Raasveldt Mark","year":"2017","unstructured":"Mark Raasveldt and Hannes M\u00fchleisen. 2017. Don't Hold My Data Hostage - A Case For Client Protocol Redesign. Proc. VLDB Endow. 10, 10 (2017), 1022\u20131033.","journal-title":"Proc. VLDB Endow."},{"key":"e_1_2_1_31_1","volume-title":"Proc. VLDB Endow. 18","author":"Schmidt Tobias","year":"2025","unstructured":"Tobias Schmidt, Viktor Leis, Peter Boncz, and Thomas Neumann. 2025. SQLStorm: Taking Database Benchmarking into the LLM Era. Proc. VLDB Endow. 18, 11 (2025)."},{"key":"e_1_2_1_32_1","volume-title":"SIGMOD Conference. 23\u201334","author":"Selinger Patricia G.","unstructured":"Patricia G. Selinger, Morton M. Astrahan, Donald D. Chamberlin, Raymond A. Lorie, and Thomas G. Price. 1979. Access Path Selection in a Relational Database Management System. In SIGMOD Conference. 23\u201334."},{"key":"e_1_2_1_33_1","unstructured":"Michael Stillger Guy M. Lohman Volker Markl and Mokhtar Kandil. 2001. LEO - DB2's LEarning Optimizer. In VLDB. 19\u201328."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.14778\/3574245.3574270"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.14778\/3229863.3236263"},{"key":"e_1_2_1_36_1","volume-title":"Leapfrog Triejoin: a worst-case optimal join algorithm. CoRR abs\/1210.0481","author":"Veldhuizen Todd L.","year":"2012","unstructured":"Todd L. Veldhuizen. 2012. Leapfrog Triejoin: a worst-case optimal join algorithm. CoRR abs\/1210.0481 (2012). http:\/\/arxiv.org\/abs\/1210.0481"},{"key":"e_1_2_1_37_1","volume-title":"Get Real: How Benchmarks Fail to Represent the Real World. In DBTest@SIGMOD. 1:1\u20131:6.","author":"Vogelsgesang Adrian","year":"2018","unstructured":"Adrian Vogelsgesang, Michael Haubenschild, Jan Finis, Alfons Kemper, Viktor Leis, Tobias M\u00fchlbauer, Thomas Neumann, and Manuel Then. 2018. Get Real: How Benchmarks Fail to Represent the Real World. In DBTest@SIGMOD. 1:1\u20131:6."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.14778\/3461535.3461552"},{"key":"e_1_2_1_39_1","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1145\/3665252.3665259","article-title":"From Binary Join to Free Join","volume":"53","author":"Wang Yisu Remy","year":"2024","unstructured":"Yisu Remy Wang, Max Willsey, and Dan Suciu. 2024. From Binary Join to Free Join. SIGMOD Rec. 53, 1 (2024), 25\u201331.","journal-title":"SIGMOD Rec."},{"key":"e_1_2_1_40_1","volume-title":"JOB-Complex: A Challenging Benchmark for Traditional & Learned Query Optimization. CoRR abs\/2507.07471","author":"Wehrstein Johannes","year":"2025","unstructured":"Johannes Wehrstein, Timo Eckmann, Roman Heinrich, and Carsten Binnig. 2025. JOB-Complex: A Challenging Benchmark for Traditional & Learned Query Optimization. CoRR abs\/2507.07471 (2025)."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.14778\/3611479.3611528"},{"key":"e_1_2_1_42_1","volume-title":"SIGMOD Conference. 2009\u20132022","author":"Wu Peizhi","year":"2021","unstructured":"Peizhi Wu and Gao Cong. 2021. A Unified Deep Model of Learning from both Data and Queries for Cardinality Estimation. In SIGMOD Conference. 2009\u20132022."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.14778\/3685800.3685818"},{"key":"e_1_2_1_44_1","first-page":"3","volume-title":"Proc. ACM Manag. Data 3","author":"Yang Yifei","year":"2025","unstructured":"Yifei Yang and Xiangyao Yu. 2025. Accelerate Distributed Joins with Predicate Transfer. Proc. ACM Manag. Data 3, 3 (2025), 122:1\u2013122:27."},{"key":"e_1_2_1_45_1","volume-title":"Predicate Transfer: Efficient Pre-Filtering on Multi-Join Queries. In CIDR.","author":"Yang Yifei","year":"2024","unstructured":"Yifei Yang, Hangdong Zhao, Xiangyao Yu, and Paraschos Koutris. 2024. Predicate Transfer: Efficient Pre-Filtering on Multi-Join Queries. In CIDR."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.14778\/3421424.3421432"},{"key":"e_1_2_1_47_1","volume-title":"Proc. VLDB Conference. VLDB Endowment, 82\u201394","author":"Yannakakis Mihalis","year":"1981","unstructured":"Mihalis Yannakakis. 1981. Algorithms for acyclic database schemes. In Proc. VLDB Conference. VLDB Endowment, 82\u201394."},{"key":"e_1_2_1_48_1","first-page":"3","volume-title":"Proc. ACM Manag. Data 3","author":"Zhang Haozhe","year":"2025","unstructured":"Haozhe Zhang, Christoph Mayer, Mahmoud Abo Khamis, Dan Olteanu, and Dan Suciu. 2025. LpBound: Pessimistic Cardinality Estimation Using \u2113p-Norms of Degree Sequences. Proc. ACM Manag. Data 3, 3 (2025), 184:1\u2013184:27."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.14778\/3611479.3611501"},{"key":"e_1_2_1_50_1","first-page":"3","volume-title":"Proc. ACM Manag. Data 3","author":"Zhao Junyi","year":"2025","unstructured":"Junyi Zhao, Kai Su, Yifei Yang, Xiangyao Yu, Paraschos Koutris, and Huanchen Zhang. 2025. Debunking the Myth of Join Ordering: Toward Robust SQL Analytics. Proc. ACM Manag. Data 3, 3 (2025), 146:1\u2013146:28."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.14778\/3090163.3090167"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3750601.3760521","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,16]],"date-time":"2025-09-16T13:43:06Z","timestamp":1758030186000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3750601.3760521"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8]]},"references-count":51,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2025,8]]}},"alternative-id":["10.14778\/3750601.3760521"],"URL":"https:\/\/doi.org\/10.14778\/3750601.3760521","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2025,8]]},"assertion":[{"value":"2025-09-16","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}