{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,26]],"date-time":"2026-08-26T19:43:50Z","timestamp":1787773430341,"version":"build-2784847793"},"reference-count":54,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,3,12]],"date-time":"2024-03-12T00:00:00Z","timestamp":1710201600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100006374","name":"Huawei Technologies","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100006374","id-type":"DOI","asserted-by":"publisher"}]},{"name":"ESPRC"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Manag. Data"],"published-print":{"date-parts":[[2024,3,12]]},"abstract":"<jats:p>Machine learning models based on neural networks (NNs) are enjoying ever-increasing attention in the Database (DB) community, both in research and practice. However, an important issue has been largely overlooked, namely the challenge of dealing with the inherent, highly dynamic nature of DBs, where data updates are fundamental, highly-frequent operations (unlike, for instance, in ML classification tasks). Although some recent research has addressed the issues of maintaining updated NN models in the presence of new data insertions, the effects of data deletions (a.k.a., \"machine unlearning\") remain a blind spot. With this work, for the first time to our knowledge, we pose and answer the following key questions: What is the effect of unlearning algorithms on NN-based DB models? How do these effects translate to effects on key downstream DB tasks, such as cardinality\/selectivity estimation (SE), approximate query processing (AQP), data generation (DG), and upstream tasks like data classification (DC)? What metrics should we use to assess the impact and efficacy of unlearning algorithms in learned DBs? Is the problem of (and solutions for) machine unlearning in DBs different from that of machine learning in DBs in the face of data insertions? Is the problem of (and solutions for) machine unlearning for DBs different from unlearning in the ML literature? what are the overhead and efficiency of unlearning algorithms (versus the naive solution of retraining from scratch)? What is the sensitivity of unlearning on batching delete operations (in order to reduce model updating overheads)? If we have a suitable unlearning algorithm (forgetting old knowledge), can we combine it with an algorithm handling data insertions (new knowledge) en route to solving the general adaptability\/updatability requirement in learned DBs in the face of both data inserts and deletes? We answer these questions using a comprehensive set of experiments, various unlearning algorithms, a variety of downstream DB tasks (such as SE, AQP, and DG), and an upstream task (DC), each with different NNs, and using a variety of metrics (model-internal, and downstream-task specific) on a variety of real datasets, making this also a first key step towards a benchmark for learned DB unlearning.<\/jats:p>","DOI":"10.1145\/3639304","type":"journal-article","created":{"date-parts":[[2024,3,26]],"date-time":"2024-03-26T18:51:32Z","timestamp":1711479092000},"page":"1-26","source":"Crossref","is-referenced-by-count":14,"title":["Machine Unlearning in Learned Databases: An Experimental Analysis"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0009-0007-5089-7759","authenticated-orcid":false,"given":"Meghdad","family":"Kurmanji","sequence":"first","affiliation":[{"name":"University of Warwick, Coventry, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2993-9674","authenticated-orcid":false,"given":"Eleni","family":"Triantafillou","sequence":"additional","affiliation":[{"name":"Google DeepMind, London, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2554-3514","authenticated-orcid":false,"given":"Peter","family":"Triantafillou","sequence":"additional","affiliation":[{"name":"University of Warwick, Coventry, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,3,26]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP40001.2021.00019"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2015.35"},{"key":"e_1_2_1_3_1","unstructured":"Edward Choi Siddharth Biswal Bradley Malin Jon Duke Walter F Stewart and Jimeng Sun. 2017. Generating multi-label discrete patient records using generative adversarial networks. In Machine learning for healthcare conference. PMLR 286--305."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3318464.3389711"},{"key":"e_1_2_1_5_1","volume-title":"Tsunami: A learned multi-dimensional index for correlated data and skewed workloads. arXiv preprint arXiv:2006.13282","author":"Ding Jialin","year":"2020","unstructured":"Jialin Ding, Vikram Nathan, Mohammad Alizadeh, and Tim Kraska. 2020b. Tsunami: A learned multi-dimensional index for correlated data and skewed workloads. arXiv preprint arXiv:2006.13282 (2020)."},{"key":"e_1_2_1_6_1","volume-title":"Foundations and Trends\u00ae in Theoretical Computer Science","volume":"9","author":"Dwork Cynthia","year":"2014","unstructured":"Cynthia Dwork, Aaron Roth, et al. 2014. The algorithmic foundations of differential privacy. Foundations and Trends\u00ae in Theoretical Computer Science, Vol. 9, 3--4 (2014), 211--407."},{"key":"e_1_2_1_7_1","volume-title":"SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation. arXiv preprint arXiv:2310.12508","author":"Fan Chongyu","year":"2023","unstructured":"Chongyu Fan, Jiancheng Liu, Yihua Zhang, Dennis Wei, Eric Wong, and Sijia Liu. 2023. SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation. arXiv preprint arXiv:2310.12508 (2023)."},{"key":"e_1_2_1_8_1","volume-title":"Fast Machine Unlearning Without Retraining Through Selective Synaptic Dampening. arXiv preprint arXiv:2308.07707","author":"Foster Jack","year":"2023","unstructured":"Jack Foster, Stefan Schoepf, and Alexandra Brintrup. 2023. Fast Machine Unlearning Without Retraining Through Selective Synaptic Dampening. arXiv preprint arXiv:2308.07707 (2023)."},{"key":"e_1_2_1_9_1","volume-title":"Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems","author":"Ginart Antonio","year":"2019","unstructured":"Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. 2019. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, Vol. 32 (2019)."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00085"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00932"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58526-6_23"},{"key":"e_1_2_1_13_1","first-page":"18932","article-title":"Revisiting deep learning models for tabular data","volume":"34","author":"Gorishniy Yury","year":"2021","unstructured":"Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. 2021. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, Vol. 34 (2021), 18932--18943.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_14_1","volume-title":"Certified data removal from machine learning models. arXiv preprint arXiv:1911.03030","author":"Guo Chuan","year":"2019","unstructured":"Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. 2019. Certified data removal from machine learning models. arXiv preprint arXiv:1911.03030 (2019)."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3318464.3389741"},{"key":"e_1_2_1_16_1","volume-title":"DeepDB: Learn from Data, not from Queries! Proceedings of the VLDB Endowment","author":"Hilprecht Benjamin","year":"2019","unstructured":"Benjamin Hilprecht, Andreas Schmidt, Moritz Kulessa, Alejandro Molina, Kristian Kersting, and Carsten Binnig. 2019. DeepDB: Learn from Data, not from Queries! Proceedings of the VLDB Endowment, Vol. 13, 7 (2019)."},{"key":"e_1_2_1_17_1","volume-title":"International Conference on Artificial Intelligence and Statistics. PMLR","author":"Izzo Zachary","year":"2021","unstructured":"Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. 2021. Approximate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics. PMLR, 2008--2016."},{"key":"e_1_2_1_18_1","volume-title":"Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems","author":"Jacot Arthur","year":"2018","unstructured":"Arthur Jacot, Franck Gabriel, and Cl\u00e9ment Hongler. 2018. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, Vol. 31 (2018)."},{"key":"e_1_2_1_19_1","volume-title":"Model Sparsity Can Simplify Machine Unlearning. In Annual Conference on Neural Information Processing Systems.","author":"Jia Jinghan","year":"2023","unstructured":"Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. 2023. Model Sparsity Can Simplify Machine Unlearning. In Annual Conference on Neural Information Processing Systems."},{"key":"e_1_2_1_20_1","volume-title":"IEEE Conference on Computer Vision & Pattern Recognition. 770--778","author":"Jian S","year":"2016","unstructured":"S Jian, H Kaiming, R Shaoqing, and Z Xiangyu. 2016. Deep residual learning for image recognition. In IEEE Conference on Computer Vision & Pattern Recognition. 770--778."},{"key":"e_1_2_1_21_1","volume-title":"Learned cardinalities: Estimating correlated joins with deep learning. arXiv preprint arXiv:1809.00677","author":"Kipf Andreas","year":"2018","unstructured":"Andreas Kipf, Thomas Kipf, Bernhard Radke, Viktor Leis, Peter Boncz, and Alfons Kemper. 2018. Learned cardinalities: Estimating correlated joins with deep learning. arXiv preprint arXiv:1809.00677 (2018)."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3196909"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3588713"},{"key":"e_1_2_1_24_1","volume-title":"Conference in Neural Information Processing Systems","author":"Kurmanji Meghdad","year":"2023","unstructured":"Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. 2023. Towards Unbounded Machine Unlearning. arXiv preprint arXiv:2302.09880, to appear in NeurIPS 2024, Conference in Neural Information Processing Systems (2023)."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3514221.3526179"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.14778\/3352063.3352129"},{"key":"e_1_2_1_27_1","volume-title":"Mehrdad Almasi, Meghdad Kurmanji, and Peter Triantafillou.","author":"Ma Qingzhi","year":"2021","unstructured":"Qingzhi Ma, Ali Mohammadi Shanghooshabad, Mehrdad Almasi, Meghdad Kurmanji, and Peter Triantafillou. 2021. Learned Approximate Query Processing: Make it Light, Accurate and Fast.. In CIDR."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3324958"},{"key":"e_1_2_1_29_1","volume-title":"Certifiable machine unlearning for linear models. arXiv preprint arXiv:2106.15093","author":"Mahadevan Ananth","year":"2021","unstructured":"Ananth Mahadevan and Michael Mathioudakis. 2021. Certifiable machine unlearning for linear models. arXiv preprint arXiv:2106.15093 (2021)."},{"key":"e_1_2_1_30_1","volume-title":"Neo: A learned query optimizer. arXiv preprint arXiv:1904.03711","author":"Marcus Ryan","year":"2019","unstructured":"Ryan Marcus, Parimarjan Negi, Hongzi Mao, Chi Zhang, Mohammad Alizadeh, Tim Kraska, Olga Papaemmanouil, and Nesime Tatbul. 2019. Neo: A learned query optimizer. arXiv preprint arXiv:1904.03711 (2019)."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3318464.3380579"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.14778\/3476249.3476259"},{"key":"e_1_2_1_33_1","volume-title":"Data synthesis based on generative adversarial networks. arXiv preprint arXiv:1806.03384","author":"Park Noseong","year":"2018","unstructured":"Noseong Park, Mahmoud Mohammadi, Kshitij Gorde, Sushil Jajodia, Hongkyu Park, and Youngmin Kim. 2018. Data synthesis based on generative adversarial networks. arXiv preprint arXiv:1806.03384 (2018)."},{"key":"e_1_2_1_34_1","first-page":"18075","article-title":"Remember what you want to forget: Algorithms for machine unlearning","volume":"34","author":"Sekhari Ayush","year":"2021","unstructured":"Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. 2021. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems, Vol. 34 (2021), 18075--18086.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3448016.3457302"},{"key":"e_1_2_1_36_1","volume-title":"DeepClean: Machine Unlearning on the Cheap by Resetting Privacy Sensitive Weights using the Fisher Diagonal. arXiv preprint arXiv:2311.10448","author":"Shi Jiaeli","year":"2023","unstructured":"Jiaeli Shi, Najah Ghalyan, Kostis Gourgoulias, John Buford, and Sean Moran. 2023. DeepClean: Machine Unlearning on the Cheap by Resetting Privacy Sensitive Weights using the Fisher Diagonal. arXiv preprint arXiv:2311.10448 (2023)."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3318464.3380584"},{"key":"e_1_2_1_38_1","volume-title":"Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806","author":"Springenberg Jost Tobias","year":"2014","unstructured":"Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. 2014. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806 (2014)."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE48307.2020.00117"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/EuroSP53844.2022.00027"},{"key":"e_1_2_1_41_1","volume-title":"31st USENIX Security Symposium (USENIX Security 22)","author":"Thudi Anvith","year":"2022","unstructured":"Anvith Thudi, Hengrui Jia, Ilia Shumailov, and Nicolas Papernot. 2022b. On the necessity of auditable algorithmic definitions for machine unlearning. In 31st USENIX Security Symposium (USENIX Security 22). 4007--4022."},{"key":"e_1_2_1_42_1","volume-title":"Peter Triantafillou, Kairan Zhao, Lisheng Sun Hosoya, Julio C. S. Jacques Junior, Vincent Dumoulin, Ioannis Mitliagkas, Sergio Escalera, Jun Wan, Sohier Dane, Maggie Demkin, and Walter Reade.","author":"Triantafillou Eleni","year":"2023","unstructured":"Eleni Triantafillou, Fabian Pedregosa, Jamie Hayes, Peter Kairouz, Isabelle Guyon, Meghdad Kurmanji, Gintare Karolina Dziugaite, Peter Triantafillou, Kairan Zhao, Lisheng Sun Hosoya, Julio C. S. Jacques Junior, Vincent Dumoulin, Ioannis Mitliagkas, Sergio Escalera, Jun Wan, Sohier Dane, Maggie Demkin, and Walter Reade. 2023. NeurIPS 2023 - Machine Unlearning. https:\/\/kaggle.com\/competitions\/neurips-2023-machine-unlearning"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3064029"},{"key":"e_1_2_1_44_1","volume-title":"Are We Ready For Learned Cardinality Estimation? arXiv preprint arXiv:2012.06743","author":"Wang Xiaoying","year":"2020","unstructured":"Xiaoying Wang, Changbo Qu, Weiyuan Wu, Jiannan Wang, and Qingqing Zhou. 2020. Are We Ready For Learned Cardinality Estimation? arXiv preprint arXiv:2012.06743 (2020)."},{"key":"e_1_2_1_45_1","volume-title":"International Conference on Machine Learning. PMLR, 10355--10366","author":"Wu Yinjun","year":"2020","unstructured":"Yinjun Wu, Edgar Dobriban, and Susan Davidson. 2020. Deltagrad: Rapid retraining of machine learning models. In International Conference on Machine Learning. PMLR, 10355--10366."},{"key":"e_1_2_1_46_1","volume-title":"FactorJoin: A New Cardinality Estimation Framework for Join Queries","author":"Wu Ziniu","year":"2023","unstructured":"Ziniu Wu, Parimarjan Negi, Alizadeh Mohammad, Tim Kraska, and Madden Samuel. 2023. FactorJoin: A New Cardinality Estimation Framework for Join Queries., Vol. 1, 1 (2023)."},{"key":"e_1_2_1_47_1","volume-title":"Advances in Neural Information Processing Systems","volume":"32","author":"Xu Lei","year":"2019","unstructured":"Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. 2019. Modeling tabular data using conditional gan. Advances in Neural Information Processing Systems, Vol. 32 (2019)."},{"key":"e_1_2_1_48_1","volume-title":"NeuroCard: one cardinality estimator for all tables. arXiv preprint arXiv:2006.08109","author":"Yang Zongheng","year":"2020","unstructured":"Zongheng Yang, Amog Kamsetty, Sifei Luan, Eric Liang, Yan Duan, Xi Chen, and Ion Stoica. 2020. NeuroCard: one cardinality estimator for all tables. arXiv preprint arXiv:2006.08109 (2020)."},{"key":"e_1_2_1_49_1","volume-title":"Deep unsupervised cardinality estimation. arXiv preprint arXiv:1905.04278","author":"Yang Zongheng","year":"2019","unstructured":"Zongheng Yang, Eric Liang, Amog Kamsetty, Chenggang Wu, Yan Duan, Xi Chen, Pieter Abbeel, Joseph M Hellerstein, Sanjay Krishnan, and Ion Stoica. 2019. Deep unsupervised cardinality estimation. arXiv preprint arXiv:1905.04278 (2019)."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3300085"},{"key":"e_1_2_1_51_1","first-page":"13433","article-title":"Prompt certified machine unlearning with randomized gradient smoothing and quantization","volume":"35","author":"Zhang Zijie","year":"2022","unstructured":"Zijie Zhang, Yang Zhou, Xin Zhao, Tianshi Che, and Lingjuan Lyu. 2022. Prompt certified machine unlearning with randomized gradient smoothing and quantization. Advances in Neural Information Processing Systems, Vol. 35 (2022), 13433--13455.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3448016.3457546"},{"key":"e_1_2_1_53_1","volume-title":"FLAT: Fast, Lightweight and Accurate Method for Cardinality Estimation. arXiv preprint arXiv:2011.09022","author":"Zhu Rong","year":"2020","unstructured":"Rong Zhu, Ziniu Wu, Yuxing Han, Kai Zeng, Andreas Pfadler, Zhengping Qian, Jingren Zhou, and Bin Cui. 2020. FLAT: Fast, Lightweight and Accurate Method for Cardinality Estimation. arXiv preprint arXiv:2011.09022 (2020)."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1186\/s13638-019-1605-z"}],"container-title":["Proceedings of the ACM on Management of Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3639304","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3639304","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,21]],"date-time":"2025-08-21T15:12:55Z","timestamp":1755789175000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3639304"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,12]]},"references-count":54,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,3,12]]}},"alternative-id":["10.1145\/3639304"],"URL":"https:\/\/doi.org\/10.1145\/3639304","relation":{},"ISSN":["2836-6573"],"issn-type":[{"value":"2836-6573","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,12]]}}}