{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,22]],"date-time":"2026-04-22T14:05:35Z","timestamp":1776866735801,"version":"3.51.2"},"reference-count":53,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,5,14]],"date-time":"2024-05-14T00:00:00Z","timestamp":1715644800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGMOD Rec."],"published-print":{"date-parts":[[2024,5,14]]},"abstract":"<jats:p>Differential privacy (DP) data synthesizers are increasingly proposed to afford public release of sensitive information, offering theoretical guarantees for privacy (and, in some cases, utility), but limited empirical evidence of utility in practical settings. Utility is typically measured as the error on representative proxy tasks, such as descriptive statistics, multivariate correlations, the accuracy of trained classifiers, or performance over a query workload. The ability for these results to generalize to practitioners' experience has been questioned in a number of settings, including the U.S. Census. In this paper, we propose an evaluation methodology for synthetic data that avoids assumptions about the representativeness of proxy tasks, instead measuring the likelihood that published conclusions would change had the authors used synthetic data, a condition we call epistemic parity. Our methodology consists of reproducing empirical conclusions of peer-reviewed papers on real, publicly available data, then re-running these experiments a second time on DP synthetic data and comparing the results.<\/jats:p>","DOI":"10.1145\/3665252.3665267","type":"journal-article","created":{"date-parts":[[2024,5,14]],"date-time":"2024-05-14T22:04:33Z","timestamp":1715724273000},"page":"65-74","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Epistemic Parity: Reproducibility as an Evaluation Metric for Differential Privacy"],"prefix":"10.1145","volume":"53","author":[{"given":"Lucas","family":"Rosenblatt","sequence":"first","affiliation":[{"name":"New York University, New York, NY, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bernease","family":"Herman","sequence":"additional","affiliation":[{"name":"University of Washington, Seattle, WA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Anastasia","family":"Holovenko","sequence":"additional","affiliation":[{"name":"Ukrainian Catholic University, Lviv, Ukraine"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wonkwon","family":"Lee","sequence":"additional","affiliation":[{"name":"New York University, New York, NY, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Joshua","family":"Loftus","sequence":"additional","affiliation":[{"name":"London School of Economics, London, United Kingdom"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Elizabeth","family":"McKinnie","sequence":"additional","affiliation":[{"name":"Microsoft, Seattle, WA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Taras","family":"Rumezhak","sequence":"additional","affiliation":[{"name":"Ukrainian Catholic University, Lviv, Ukraine"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Andrii","family":"Stadnik","sequence":"additional","affiliation":[{"name":"Ukrainian Catholic University, Lviv, Ukraine"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bill","family":"Howe","sequence":"additional","affiliation":[{"name":"University of Washington, Seattle, WA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Julia","family":"Stoyanovich","sequence":"additional","affiliation":[{"name":"New York University, New York, NY, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,5,14]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.3390\/ijerph16193705"},{"key":"e_1_2_1_2_1","volume-title":"Differential privacy has disparate impact on model accuracy. Advances in neural information processing systems, 32","author":"Bagdasaryan E.","year":"2019","unstructured":"E. Bagdasaryan, O. Poursaeed, and V. Shmatikov. Differential privacy has disparate impact on model accuracy. Advances in neural information processing systems, 32, 2019."},{"key":"e_1_2_1_3_1","volume-title":"Nature, 533(7604)","author":"Baker M.","year":"2016","unstructured":"M. Baker. 1,500 scientists lift the lid on reproducibility. Nature, 533(7604), 2016."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1214\/19-STS742"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1162\/99608f92.66882f0e"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-662-53641-4_24"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.14778\/3476249.3476272"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP46214.2022.9833668"},{"key":"e_1_2_1_9_1","first-page":"156","volume-title":"International Conference on Language Resources & Evaluation:[proceedings]. International Conference on Language Resources and Evaluation","volume":"2018","author":"Cohen K. B.","unstructured":"K. B. Cohen, J. Xia, P. Zweigenbaum, T. J. Callahan, O. Hargraves, F. Goss, N. Ide, A. N\u00b4ev\u00b4eol, C. Grouin, and L. E. Hunter. Three dimensions of reproducibility in natural language processing. In LREC... International Conference on Language Resources & Evaluation:[proceedings]. International Conference on Language Resources and Evaluation, volume 2018, page 156. NIH Public Access, 2018."},{"key":"e_1_2_1_11_1","volume-title":"NeurIPS","author":"Ding F.","year":"2021","unstructured":"F. Ding, M. Hardt, J. Miller, and L. Schmidt. Retiring adult: New datasets for fair machine learning. NeurIPS, 2021."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i01.5402"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/773153.773173"},{"key":"e_1_2_1_14_1","volume-title":"UCI machine learning repository","author":"Dua D.","year":"2017","unstructured":"D. Dua and C. Graff. UCI machine learning repository, 2017."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1007\/11681878_14"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1536414.1536467"},{"key":"e_1_2_1_17_1","volume-title":"The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3--4):211--407","author":"Dwork C.","year":"2014","unstructured":"C. Dwork, A. Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3--4):211--407, 2014."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.67995"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11121-018-0908-3"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1002\/ajcp.12233"},{"key":"e_1_2_1_21_1","volume-title":"Robin hood and matthew effects--differential privacy has disparate impact on synthetic data. arXiv preprint arXiv:2109.11429","author":"Ganev G.","year":"2021","unstructured":"G. Ganev, B. Oprisanu, and E. De Cristofaro. Robin hood and matthew effects--differential privacy has disparate impact on synthetic data. arXiv preprint arXiv:2109.11429, 2021."},{"key":"e_1_2_1_22_1","volume-title":"A simple and practical algorithm for differentially private data release. arXiv preprint arXiv:1012.4763","author":"Hardt M.","year":"2010","unstructured":"M. Hardt, K. Ligett, and F. McSherry. A simple and practical algorithm for differentially private data release. arXiv preprint arXiv:1012.4763, 2010."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.3886\/ICPSR21600.v25"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2882931"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-23633-9_4"},{"key":"e_1_2_1_27_1","volume-title":"High school football and risk for depression and suicidality in adulthood: findings from a national longitudinal study. Frontiers in neurology, 12","author":"Iverson G. L.","year":"2021","unstructured":"G. L. Iverson and D. P. Terry. High school football and risk for depression and suicidality in adulthood: findings from a national longitudinal study. Frontiers in neurology, 12, 2021."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1214\/ss\/1177013012"},{"key":"e_1_2_1_29_1","first-page":"3000","volume-title":"ICML","author":"Jagielski M.","year":"2019","unstructured":"M. Jagielski, M. Kearns, J. Mao, A. Oprea, A. Roth, S. Sharifi-Malvajerdi, and J. Ullman. Differentially private fair learning. In ICML, pages 3000--3008, 2019."},{"key":"e_1_2_1_30_1","volume-title":"35th Conference on Neural Information Processing Systems (NeurIPS 2021) Workshop on Math AI for Education (MATHAI4ED)","author":"Jeong H.","year":"2021","unstructured":"H. Jeong, M. D. Wu, N. Dasgupta, M. M\u00b4edard, and F. P. Calmon. Whogets the benefit of the doubt? racial bias in machine learning algorithms applied to secondary school math education. In 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Workshop on Math AI for Education (MATHAI4ED), 2021."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1111\/1467-9884.00122"},{"key":"e_1_2_1_32_1","volume-title":"The use of differential privacy for census data and its impact on redistricting: The case of the 2020 us census. Science advances, 7(41):eabk3283","author":"Kenny C. T.","year":"2021","unstructured":"C. T. Kenny, S. Kuriwaki, C. McCartan, E. T. Rosenman, T. Simko, and K. Imai. The use of differential privacy for census data and its impact on redistricting: The case of the 2020 us census. Science advances, 7(41):eabk3283, 2021."},{"key":"e_1_2_1_33_1","volume-title":"UCI Machine Learning Repository","author":"Kohavi R.","year":"1996","unstructured":"R. Kohavi and B. Becker. UCI adult data set. Technical report, UCI Machine Learning Repository, 1996. https:\/\/archive.ics.uci.edu\/ml\/datasets\/adult."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.adolescence.2021.01.008"},{"key":"e_1_2_1_35_1","first-page":"690","article-title":"Iterative methods for private synthetic data: Unifying framework and new methods","volume":"34","author":"Liu T.","year":"2021","unstructured":"T. Liu, G. Vietri, and S. Z. Wu. Iterative methods for private synthetic data: Unifying framework and new methods. Advances in Neural Information Processing Systems, 34:690--702, 2021.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_36_1","volume-title":"Optimizing error of high-dimensional statistical queries under differential privacy. arXiv preprint arXiv:1808.03537","author":"McKenna R.","year":"2018","unstructured":"R. McKenna, G. Miklau, M. Hay, and A. Machanavajjhala. Optimizing error of high-dimensional statistical queries under differential privacy. arXiv preprint arXiv:1808.03537, 2018."},{"key":"e_1_2_1_37_1","volume-title":"Winning the NIST contest: A scalable and general approach to differentially private synthetic data. arXiv preprint arXiv:2108.04978","author":"McKenna R.","year":"2021","unstructured":"R. McKenna, G. Miklau, and D. Sheldon. Winning the NIST contest: A scalable and general approach to differentially private synthetic data. arXiv preprint arXiv:2108.04978, 2021."},{"key":"e_1_2_1_38_1","volume-title":"Aim: An adaptive and iterative mechanism for differentially private synthetic data. arXiv preprint arXiv:2201.12677","author":"McKenna R.","year":"2022","unstructured":"R. McKenna, B. Mullins, D. Sheldon, and G. Miklau. Aim: An adaptive and iterative mechanism for differentially private synthetic data. arXiv preprint arXiv:2201.12677, 2022."},{"key":"e_1_2_1_39_1","volume-title":"Engineering, and Medicine and others. Reproducibility and replicability in science","author":"National Academies of Sciences","year":"2019","unstructured":"National Academies of Sciences, Engineering, and Medicine and others. Reproducibility and replicability in science. 2019."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1177\/0265407518817400"},{"key":"e_1_2_1_41_1","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1007\/978-3-319-31753-3_18","volume-title":"Advances in Knowledge Discovery and Data Mining","author":"Pinto F.","year":"2016","unstructured":"F. Pinto, C. Soares, and J. Mendes-Moreira. Towards automatic generation of metafeatures. In J. Bailey, L. Khan, T. Washio, G. Dobbie, J. Z. Huang, and R. Wang, editors, Advances in Knowledge Discovery and Data Mining, pages 215--226, Cham, 2016. Springer International Publishing."},{"key":"e_1_2_1_42_1","volume-title":"de Leon Ferreira de Carvalho. Characterizing classification datasets: a study of meta-features for meta-learning. arXiv: Learning","author":"Rivolli A.","year":"2018","unstructured":"A. Rivolli, L. P. F. Garcia, C. Soares, J. Vanschoren, and A. C. P. de Leon Ferreira de Carvalho. Characterizing classification datasets: a study of meta-features for meta-learning. arXiv: Learning, 2018."},{"key":"e_1_2_1_43_1","volume-title":"Spending privacy budget fairly and wisely. CoRR, abs\/2204.12903","author":"Rosenblatt L.","year":"2022","unstructured":"L. Rosenblatt, J. Allen, and J. Stoyanovich. Spending privacy budget fairly and wisely. CoRR, abs\/2204.12903, 2022."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.14778\/3611479.3611517"},{"key":"e_1_2_1_45_1","volume-title":"Differentially private synthetic data: Applied evaluations and enhancements. arXiv preprint arXiv:2011.05537","author":"Rosenblatt L.","year":"2020","unstructured":"L. Rosenblatt, X. Liu, S. Pouyanfar, E. de Leon, A. Desai, and J. Allen. Differentially private synthetic data: Applied evaluations and enhancements. arXiv preprint arXiv:2011.05537, 2020."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.86.3.638"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1257\/pandp.20191107"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.3102\/0013189X18787818"},{"key":"e_1_2_1_49_1","volume-title":"UCI Machine Learning Repository","author":"Schlimmer J.","year":"1987","unstructured":"J. Schlimmer. UCI adult data set. Technical report, UCI Machine Learning Repository, 1987. https: \/\/archive.ics.uci.edu\/ml\/datasets\/mushroom."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE51399.2021.00022"},{"key":"e_1_2_1_51_1","volume-title":"Benchmarking differentially private synthetic data generation algorithms. arXiv preprint arXiv:2112.09238","author":"Tao Y.","year":"2021","unstructured":"Y. Tao, R. McKenna, M. Hay, A. Machanavajjhala, and G. Miklau. Benchmarking differentially private synthetic data generation algorithms. arXiv preprint arXiv:2112.09238, 2021."},{"key":"e_1_2_1_52_1","volume-title":"DP-CGAN: differentially private synthetic data and label generation. CoRR, abs\/2001.09700","author":"Torkzadehmahani R.","year":"2020","unstructured":"R. Torkzadehmahani, P. Kairouz, and B. Paten. DP-CGAN: differentially private synthetic data and label generation. CoRR, abs\/2001.09700, 2020."},{"key":"e_1_2_1_54_1","first-page":"9765","volume-title":"ICML","author":"Vietri G.","year":"2020","unstructured":"G. Vietri, G. Tian, M. Bun, T. Steinke, and S. Wu. New oracle-efficient algorithms for private synthetic data release. In ICML, pages 9765--9774, 2020."},{"key":"e_1_2_1_55_1","first-page":"32","article-title":"Modeling tabular data using conditional gan","author":"Xu L.","year":"2019","unstructured":"L. Xu, M. Skoularidou, A. Cuesta-Infante, and K. Veeramachaneni. Modeling tabular data using conditional gan. Advances in Neural Information Processing Systems, 32, 2019.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2588573"}],"container-title":["ACM SIGMOD Record"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3665252.3665267","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3665252.3665267","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:58:33Z","timestamp":1750294713000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3665252.3665267"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,14]]},"references-count":53,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,5,14]]}},"alternative-id":["10.1145\/3665252.3665267"],"URL":"https:\/\/doi.org\/10.1145\/3665252.3665267","relation":{},"ISSN":["0163-5808"],"issn-type":[{"value":"0163-5808","type":"print"}],"subject":[],"published":{"date-parts":[[2024,5,14]]},"assertion":[{"value":"2024-05-14","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}