{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,24]],"date-time":"2026-02-24T16:51:49Z","timestamp":1771951909804,"version":"3.50.1"},"reference-count":36,"publisher":"Association for Computing Machinery (ACM)","issue":"11","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2025,7]]},"abstract":"<jats:p>Existing research on the accuracy of cardinality estimators generally suffers from a lack of diversity and sufficient quantity of their experimental datasets, particularly in relation to the claimed scope of the study and the generality of its conclusions. We argue that a sufficiently large number of varied datasets are essential for comprehensive evaluations. However, the prevailing per-dataset evaluation method (PDE), producing one result table per dataset, has so far hindered this necessary expansion of the experiments. Moreover, as we demonstrate, this evaluation method often leaves the reader with contradictory results, where one estimator excels on certain datasets or queries, while the other performs better elsewhere. To address these and similar limitations, we propose a multidimensional evaluation framework. This framework unravels the conundrum of analyzing the evaluation results across multiple datasets through the use of discretization. It establishes a robust foundation for aggregating the evaluation results and conducting pairwise comparisons between estimators. Furthermore, it facilitates informed decision making in the presence of conflicting results through a customizable ranking mechanism.<\/jats:p>\n          <jats:p>To empirically highlight the shortcomings of the aforementioned per-dataset evaluation and demonstrate the advantages of our proposed framework, we conduct a benchmarking study of cardinality estimators, incorporating both learned and traditional approaches. We focus on a fundamental challenge: estimating the cardinality of range queries on a single 2-D geographical relation in a static environment. Despite the apparent simplicity of this task, our findings reveal that many estimators struggle to handle this challenge effectively. To further enhance the quality of our study, we provide valuable insights by addressing some critical aspects that were overlooked in previous benchmarking studies.<\/jats:p>","DOI":"10.14778\/3749646.3749651","type":"journal-article","created":{"date-parts":[[2025,9,4]],"date-time":"2025-09-04T17:55:06Z","timestamp":1757008506000},"page":"3744-3756","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["The Accuracy of Cardinality Estimators: Unraveling the Evaluation Result Conundrum"],"prefix":"10.14778","volume":"18","author":[{"given":"Nazanin","family":"Rashedi","sequence":"first","affiliation":[{"name":"University of Mannheim, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Guido","family":"Moerkotte","sequence":"additional","affiliation":[{"name":"University of Mannheim, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,9,4]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"andreaskipf\/learnedcardinalities. https:\/\/github.com\/andreaskipf\/learnedcardinalities Retrieved","author":"Kipf Andreas","year":"2024","unstructured":"Andreas Kipf. 2019. andreaskipf\/learnedcardinalities. https:\/\/github.com\/andreaskipf\/learnedcardinalities Retrieved September 3, 2024."},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the International Conference on Scientific and Statistical Database Management (SSDBM).","author":"Buccafurri F.","unstructured":"F. Buccafurri, F. Furfaro, G. M. Mazzeo, and D. Sacca. 2003. A quad-tree based multiresolution approach for two-dimensional summary data. In Proceedings of the International Conference on Scientific and Statistical Database Management (SSDBM)."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939785"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1015330.1015408"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.14778\/3329772.3329780"},{"key":"e_1_2_1_6_1","unstructured":"L\u00e9o Grinsztajn Edouard Oyallon and Ga\u00ebl Varoquaux. 2022. Why do tree-based models still outperform deep learning on tabular data? arXiv:2207.08815 [cs.LG]"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/602259.602266"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.14778\/3503585.3503586"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.14778\/3384345.3384349"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.14778\/2994509.2994529"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3639300"},{"key":"e_1_2_1_12_1","volume-title":"Learned Cardinalities: Estimating Correlated Joins with Deep Learning. arXiv:1809.00677 [cs]","author":"Kipf Andreas","year":"2018","unstructured":"Andreas Kipf, Thomas Kipf, Bernhard Radke, Viktor Leis, Peter Boncz, and Alfons Kemper. 2018. Learned Cardinalities: Estimating Correlated Joins with Deep Learning. arXiv:1809.00677 [cs]. http:\/\/arxiv.org\/abs\/1809.00677"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1201\/b16923"},{"key":"e_1_2_1_14_1","unstructured":"Data Management Lab. 2020. DeepDB: Deep Learning for Query Processing. https:\/\/github.com\/DataManagementLab\/deepdb-public. Accessed: 2024-09-03."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.14778\/3611479.3611494"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.14778\/2850583.2850594"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2408.17209"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3639270"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11280-023-01195-7"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.5555\/602876.602909"},{"key":"e_1_2_1_21_1","unstructured":"Paulius Micikevicius Sharan Narang Jonah Alben Gregory Diamos Erich Elsen David Garcia Boris Ginsburg Michael Houston Oleksii Kuchaiev Ganesh Venkatesh and Hao Wu. 2018. Mixed Precision Training. arXiv:1710.03740 [cs.AI]."},{"key":"e_1_2_1_22_1","unstructured":"Guido Moerkotte. 2024. Query Compiler. https:\/\/pi3.informatik.uni-mannheim.de\/~moer\/querycompiler.pdf. Accessed: 2024-07-16."},{"key":"e_1_2_1_23_1","volume-title":"June 22, 2017. (accessed 2024-07-16)","author":"Moerkotte Guido","year":"2017","unstructured":"Guido Moerkotte, Norman May, and Anja Boehm. 2017. Methods and Systems for Estimating the Number of Points in Two-Dimensional Data. https:\/\/patents.google.com\/patent\/US20170177663A1\/en_US20170177663A1, June 22, 2017. (accessed 2024-07-16)."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.14778\/1687627.1687738"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/971701.50205"},{"key":"e_1_2_1_26_1","volume-title":"naru-project\/naru. https:\/\/github.com\/naru-project\/naru Retrieved","year":"2024","unstructured":"naru-project. 2019. naru-project\/naru. https:\/\/github.com\/naru-project\/naru Retrieved September 3, 2024."},{"key":"e_1_2_1_27_1","unstructured":"OpenStreetMap Foundation. n.d.. OpenStreetMap. https:\/\/www.openstreet.map.org Accessed: 2024-08-31."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.5555\/3020548.3020596"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/971701.50205"},{"key":"e_1_2_1_30_1","unstructured":"PostgreSQL Global Development Group. 2023. Statistics Used by the Planner. https:\/\/www.postgresql.org\/docs\/15\/planner-stats.html Accessed: 2024-06-11."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.14778\/3485450.3485459"},{"key":"e_1_2_1_32_1","unstructured":"U.S. Census Bureau. 2023. TIGER\/Line Shapefiles and TIGER\/Line Geo-database. https:\/\/www.census.gov\/geographies\/mapping-files\/time-series\/geo\/tiger-geodatabase-file.html Accessed: 2024-08-31."},{"key":"e_1_2_1_33_1","unstructured":"U.S. Geological Survey. 2013. Earthquake Data Feed. https:\/\/earthquake.usgs.gov\/earthquakes\/feed\/v1.0\/csv.php Accessed: 2024-08-31."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.14778\/3461535.3461552"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3329859.3329875"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.14778\/3368289.3368294"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3749646.3749651","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,5]],"date-time":"2025-09-05T03:17:29Z","timestamp":1757042249000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3749646.3749651"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7]]},"references-count":36,"journal-issue":{"issue":"11","published-print":{"date-parts":[[2025,7]]}},"alternative-id":["10.14778\/3749646.3749651"],"URL":"https:\/\/doi.org\/10.14778\/3749646.3749651","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2025,7]]},"assertion":[{"value":"2025-09-04","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}