{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,25]],"date-time":"2026-07-25T16:39:12Z","timestamp":1784997552982,"version":"3.55.0"},"reference-count":73,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,3,12]],"date-time":"2024-03-12T00:00:00Z","timestamp":1710201600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Manag. Data"],"published-print":{"date-parts":[[2024,3,12]]},"abstract":"<jats:p>Missing data is a widespread problem in many domains, creating challenges in data analysis and decision making. Traditional techniques for dealing with missing data, such as excluding incomplete records or imputing simple estimates (e.g., mean), are computationally efficient but may introduce bias and disrupt variable relationships, leading to inaccurate analyses. Model-based imputation techniques offer a more robust solution that preserves the variability and relationships in the data, but they demand significantly more computation time, limiting their applicability to small datasets.<\/jats:p>\n          <jats:p>This work enables efficient, high-quality, and scalable data imputation within a database system using the widely used MICE method. We adapt this method to exploit computation sharing and a ring abstraction for faster model training. To impute both continuous and categorical values, we develop techniques for in-database learning of stochastic linear regression and Gaussian discriminant analysis models. Our MICE implementations in PostgreSQL and DuckDB outperform alternative MICE implementations and model-based imputation techniques by up to two orders of magnitude in terms of computation time, while maintaining high imputation quality.<\/jats:p>","DOI":"10.1145\/3639326","type":"journal-article","created":{"date-parts":[[2024,3,26]],"date-time":"2024-03-26T18:51:32Z","timestamp":1711479092000},"page":"1-27","source":"Crossref","is-referenced-by-count":6,"title":["In-Database Data Imputation"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5006-9979","authenticated-orcid":false,"given":"Massimo","family":"Perini","sequence":"first","affiliation":[{"name":"University of Edinburgh, Edinburgh, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1548-6803","authenticated-orcid":false,"given":"Milos","family":"Nikolic","sequence":"additional","affiliation":[{"name":"University of Edinburgh, Edinburgh, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,3,26]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.14778\/2994509.2994518"},{"key":"e_1_2_2_2_1","unstructured":"Taiwan Environmental Protection Administration. 2016 - 2021. Taiwan Air Quality Data. https:\/\/data.gov.tw\/dataset\/40448"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1177\/0962280206074466"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1751--5823.2010.00103.x"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/SUPERC.1990.129995"},{"key":"e_1_2_2_6_1","unstructured":"Apache MADlib. 2023. https:\/\/madlib.apache.org."},{"key":"e_1_2_2_7_1","unstructured":"Apache SystemDS. 2023. https:\/\/systemds.apache.org."},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.14778\/3476249.3476300"},{"key":"e_1_2_2_9_1","unstructured":"Samyadeep Basu Phillip Pope and Soheil Feizi. 2021. Influence Functions in Deep Learning Are Fragile. In ICLR."},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.14778\/3204028.3204032"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","unstructured":"Felix Biessmann David Salinas Sebastian Schelter Philipp Schmidt and Dustin Lange. 2018. \"Deep\" Learning for Missing Value Imputationin Tables with Non-Numerical Data. In CIKM. 2017--2025. https:\/\/doi.org\/10.1145\/3269206.3272005","DOI":"10.1145\/3269206.3272005"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.14778\/3007263.3007279"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1201\/9780429492259"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.14778\/3137628.3137641"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.14778\/3137628.3137633"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2912574"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.14778\/3352063.3352066"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1186\/s40537-021-00516--9"},{"key":"e_1_2_2_19_1","doi-asserted-by":"crossref","unstructured":"Xixuan Feng Arun Kumar Benjamin Recht and Christopher R\u00e9. 2012. Towards a Unified Architecture for In-RDBMS Analytics. In SIGMOD. 325--336.","DOI":"10.1145\/2213836.2213874"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.psych.58.110405.085530"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.14778\/2367502.2367510"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","unstructured":"Benjamin Hilprecht and Carsten Binnig. 2021. ReStore -- Neural Data Completion for Relational Databases. In SIGMOD. 710--722. https:\/\/doi.org\/10.1145\/3448016.3457264","DOI":"10.1145\/3448016.3457264"},{"key":"e_1_2_2_23_1","doi-asserted-by":"crossref","unstructured":"Zezhou Huang Rathijit Sen Jiaxiang Liu and Eugene Wu. 2023. JoinBoost: Grow Trees Over Normalized Data Using Only SQL. arxiv: 2307.00422 [cs.DB]","DOI":"10.14778\/3611479.3611509"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-018-0615--6"},{"key":"e_1_2_2_25_1","unstructured":"Daniel Jarrett Bogdan Cebere Tennison Liu Alicia Curth and Mihaela van der Schaar. 2022. HyperImpute: Generalized Iterative Imputation with Automatic Model Selection. In ICML. 9916--9937."},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.3389\/fdata.2021.693674"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.14778\/3430915.3430917"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","unstructured":"Mahmoud Abo Khamis Hung Q. Ngo XuanLong Nguyen Dan Olteanu and Maximilian Schleich. 2018. AC\/DC: In-Database Learning Thunderstruck. In DEEM. 1--10. https:\/\/doi.org\/10.1145\/3209889.3209896","DOI":"10.1145\/3209889.3209896"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.14778\/3430915.3430920"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.14778\/3377369.3377383"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","unstructured":"Christoph Koch. 2010. Incremental Query Evaluation in a Ring of Databases. In PODS. 87--98. https:\/\/doi.org\/10.1145\/1807085.1807100","DOI":"10.1145\/1807085.1807100"},{"key":"e_1_2_2_32_1","first-page":"59","article-title":"SampleClean: Fast and Reliable Analytics on Dirty Data","volume":"38","author":"Krishnan S.","year":"2015","unstructured":"S. Krishnan, Jiannan Wang, M. Franklin, Ken Goldberg, T. Kraska, T. Milo, and E. Wu. 2015. SampleClean: Fast and Reliable Analytics on Dirty Data. IEEE Data Eng. Bull., Vol. 38 (2015), 59--75.","journal-title":"IEEE Data Eng. Bull."},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","unstructured":"Arun Kumar Matthias Boehm and Jun Yang. 2017. Data Management in Machine Learning: Challenges Techniques and Systems. In SIGMOD. 1717--1722. https:\/\/doi.org\/10.1145\/3035918.3054775","DOI":"10.1145\/3035918.3054775"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2723713"},{"key":"e_1_2_2_35_1","first-page":"23806","article-title":"MIRACLE: Causally-Aware Imputation via Learning Missing Data Mechanisms","volume":"34","author":"Kyono Trent","year":"2021","unstructured":"Trent Kyono, Yao Zhang, Alexis Bellot, and Mihaela van der Schaar. 2021. MIRACLE: Causally-Aware Imputation via Learning Missing Data Mechanisms. Advances in Neural Information Processing Systems, Vol. 34 (2021), 23806--23817.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1177\/1740774515602688"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1017\/pan.2020.49"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","unstructured":"P.-A. Larson. 2002. Data Reduction by Partial Preaggregation. In ICDT. 706--715. https:\/\/doi.org\/10.1109\/ICDE.2002.994787","DOI":"10.1109\/ICDE.2002.994787"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3318464.3389785"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1002\/9781119482260"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.14778\/3450980.3450989"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.14778\/3407790.3407801"},{"key":"e_1_2_2_43_1","volume-title":"MIWAE: Deep Generative Modelling and Imputation of Incomplete Data Sets. In ICML. 4413--4423.","author":"Mattei Pierre-Alexandre","year":"2019","unstructured":"Pierre-Alexandre Mattei and Jes Frellsen. 2019. MIWAE: Deep Generative Modelling and Imputation of Incomplete Data Sets. In ICML. 4413--4423."},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.5555\/1756006.1859931"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0254720"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.14778\/3494124.3494143"},{"key":"e_1_2_2_47_1","unstructured":"MindsDB. 2023. https:\/\/mindsdb.com\/."},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","unstructured":"Mashaal Musleh Mourad Ouzzani Nan Tang and AnHai Doan. 2020. CoClean: Collaborative Data Cleaning. In SIGMOD. 2757--2760. https:\/\/doi.org\/10.1145\/3318464.3384698","DOI":"10.1145\/3318464.3384698"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","unstructured":"Milos Nikolic and Dan Olteanu. 2018. Incremental View Maintenance with Triple Lock Factorization Benefits. In SIGMOD. 365--380. https:\/\/doi.org\/10.1145\/3183713.3183758","DOI":"10.1145\/3183713.3183758"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","unstructured":"Milos Nikolic Haozhe Zhang Ahmet Kara and Dan Olteanu. 2020. F-IVM: Learning over Fast-Evolving Relational Data. In SIGMOD. 2773--2776. https:\/\/doi.org\/10.1145\/3318464.3384702","DOI":"10.1145\/3318464.3384702"},{"key":"e_1_2_2_51_1","unstructured":"U.S. Department of Transportation. 2015. Flight Delays and Cancellations. https:\/\/www.kaggle.com\/datasets\/usdot\/flight-delays"},{"key":"e_1_2_2_52_1","article-title":"Scikit-learn: Machine Learning in Python","volume":"12","author":"Pedregosa Fabian","year":"2011","unstructured":"Fabian Pedregosa, Ga\u00ebl Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and \u00c9douard Duchesnay. 2011. Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res., Vol. 12, null (2011), 2825--2830.","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_2_2_53_1","first-page":"35839","article-title":"Missing Data Imputation and Acquisition with Deep Hierarchical Models and Hamiltonian Monte Carlo","volume":"35","author":"Peis Ignacio","year":"2022","unstructured":"Ignacio Peis, Chao Ma, and Jos\u00e9 Miguel Hern\u00e1ndez-Lobato. 2022. Missing Data Imputation and Acquisition with Deep Hierarchical Models and Hamiltonian Monte Carlo. Advances in Neural Information Processing Systems, Vol. 35 (2022), 35839--35851.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.14778\/3570690.3570694"},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","unstructured":"Jose Picado John Davis Arash Termehchy and Ga Young Lee. 2020. Learning Over Dirty Data Without Cleaning. In SIGMOD. 1301--1316. https:\/\/doi.org\/10.1145\/3318464.3389708","DOI":"10.1145\/3318464.3389708"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3220109"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.14778\/3137628.3137631"},{"key":"e_1_2_2_58_1","doi-asserted-by":"crossref","unstructured":"Weilong Ren Xiang Lian and Kambiz Ghazinour. 2021. Online Topic-Aware Entity Resolution Over Incomplete Data Streams. In SIGMOD. 1478--1490.","DOI":"10.1145\/3448016.3457238"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.14778\/3476249.3476301"},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1037\/1082--989X.7.2.147"},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.14778\/3229863.3229867"},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.14778\/3415478.3415515"},{"key":"e_1_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3324961"},{"key":"e_1_2_2_64_1","doi-asserted-by":"publisher","unstructured":"Maximilian Schleich Dan Olteanu and Radu Ciucanu. 2016. Learning Linear Regression Models over Factorized Joins. In SIGMOD. 3--18. https:\/\/doi.org\/10.1145\/2882903.2882939","DOI":"10.1145\/2882903.2882939"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btr597"},{"key":"e_1_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.14778\/3415478.3415570"},{"key":"e_1_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.14778\/3551793.3551818"},{"key":"e_1_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v045.i03"},{"key":"e_1_2_2_69_1","first-page":"5689","article-title":"GAIN: Missing Data Imputation using Generative Adversarial Nets","volume":"80","author":"Yoon Jinsung","year":"2018","unstructured":"Jinsung Yoon, James Jordon, and Mihaela van der Schaar. 2018. GAIN: Missing Data Imputation using Generative Adversarial Nets. In ICML, Vol. 80. 5689--5698.","journal-title":"ICML"},{"key":"e_1_2_2_70_1","volume-title":"GAMIN: Generative Adversarial Multiple Imputation Network for Highly Missing Data. In CVPR. 8453--8461.","author":"Yoon Seongwook","year":"2020","unstructured":"Seongwook Yoon and Sanghoon Sull. 2020. GAMIN: Generative Adversarial Multiple Imputation Network for Highly Missing Data. In CVPR. 8453--8461."},{"key":"e_1_2_2_71_1","first-page":"19075","article-title":"Handling Missing Data with Graph Representation Learning","volume":"33","author":"You Jiaxuan","year":"2020","unstructured":"Jiaxuan You, Xiaobai Ma, Yi Ding, Mykel J Kochenderfer, and Jure Leskovec. 2020. Handling Missing Data with Graph Representation Learning. Advances in Neural Information Processing Systems, Vol. 33 (2020), 19075--19087.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_2_72_1","volume-title":"Multiple Imputation for Missing Data: Concepts and New Development (Version 9.0)","author":"Yuan Yang C","year":"2010","unstructured":"Yang C Yuan. 2010. Multiple Imputation for Missing Data: Concepts and New Development (Version 9.0). SAS Institute Inc, Rockville, MD, Vol. 49, 1--11 (2010), 12."},{"key":"e_1_2_2_73_1","volume-title":"Diffusion Models for Missing Value Imputation in Tabular Data. In NeurIPS 2022 First Table Representation Workshop.","author":"Zheng Shuhan","year":"2022","unstructured":"Shuhan Zheng and Nontawat Charoenphakdee. 2022. Diffusion Models for Missing Value Imputation in Tabular Data. In NeurIPS 2022 First Table Representation Workshop."}],"container-title":["Proceedings of the ACM on Management of Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3639326","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3639326","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,21]],"date-time":"2025-08-21T15:17:04Z","timestamp":1755789424000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3639326"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,12]]},"references-count":73,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,3,12]]}},"alternative-id":["10.1145\/3639326"],"URL":"https:\/\/doi.org\/10.1145\/3639326","relation":{},"ISSN":["2836-6573"],"issn-type":[{"value":"2836-6573","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,12]]}}}