{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,10]],"date-time":"2026-10-10T05:38:37Z","timestamp":1791610717660,"version":"4.3.4"},"reference-count":40,"publisher":"Association for Computing Machinery (ACM)","issue":"7","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Commun. ACM"],"published-print":{"date-parts":[[2026,7,1]]},"abstract":"<jats:p>Differential privacy (DP) data synthesizers are increasingly proposed to afford public release of sensitive information, offering theoretical guarantees for privacy (and, in some cases, utility), but limited empirical evidence of utility in practical settings. Utility is typically measured as the error on representative proxy tasks, such as descriptive statistics, multivariate correlations, the accuracy of trained classifiers, or performance over a query workload. The ability of these results to generalize to practitioners\u2019 experience has been questioned in a number of settings, including the U.S. Census. In this paper, we propose an evaluation methodology for synthetic data that avoids assumptions about the representativeness of proxy tasks, instead measuring the likelihood that published conclusions would change had the authors used synthetic data, a condition we call epistemic parity. Our methodology consists of reproducing empirical conclusions of peer-reviewed papers on real, publicly available data, then re-running these experiments a second time on DP synthetic data and comparing the results.<\/jats:p>\n                  <jats:p>We instantiate our methodology over a benchmark of recent peer-reviewed papers in the social sciences. We express the authors\u2019 claims computationally to automate the experiment, generate DP synthetic datasets using multiple state-of-the-art mechanisms, then estimate the likelihood that these conclusions will hold. We find that, for reasonable privacy regimes, DP synthesizers can achieve high epistemic parity for several papers in our benchmark. However, some papers, and particularly some specific findings, are difficult to reproduce for any of the synthesizers. Given these results, we recommend a new class of mechanisms that offer stronger utility guarantees (as measured by epistemic parity) and more nuanced privacy protection using application-specific risks and threat models.<\/jats:p>","DOI":"10.1145\/3807967","type":"journal-article","created":{"date-parts":[[2026,6,25]],"date-time":"2026-06-25T14:46:29Z","timestamp":1782398789000},"page":"93-102","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Epistemic Parity: Reproducibility as an Evaluation Metric for Differential Privacy"],"prefix":"10.1145","volume":"69","author":[{"given":"Lucas","family":"Rosenblatt","sequence":"first","affiliation":[{"name":"New York University, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bernease","family":"Herman","sequence":"additional","affiliation":[{"name":"University of Washington, Seattle, WA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anastasia","family":"Holovenko","sequence":"additional","affiliation":[{"name":"Ukrainian Catholic University, Lviv, Ukraine"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wonkwon","family":"Lee","sequence":"additional","affiliation":[{"name":"New York University, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joshua","family":"Loftus","sequence":"additional","affiliation":[{"name":"London School of Economics, London, England, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elizabeth","family":"McKinnie","sequence":"additional","affiliation":[{"name":"University of Colorado, Boulder, CO, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Taras","family":"Rumezhak","sequence":"additional","affiliation":[{"name":"Ukrainian Catholic University, Lviv, Ukraine"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrii","family":"Stadnik","sequence":"additional","affiliation":[{"name":"Ukrainian Catholic University, Lviv, Ukraine"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bill","family":"Howe","sequence":"additional","affiliation":[{"name":"University of Washington, Seattle, WA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Julia","family":"Stoyanovich","sequence":"additional","affiliation":[{"name":"New York University, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6,30]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.3390\/ijerph16193705"},{"key":"e_1_3_1_3_2","article-title":"Differential privacy has disparate impact on model accuracy","volume":"32","author":"Bagdasaryan E.","year":"2019","unstructured":"Bagdasaryan, E., Poursaeed, O., and Shmatikov, V. Differential privacy has disparate impact on model accuracy. Advances in Neural Information Processing Systems\u00a032, (2019).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.1214\/19-STS742"},{"key":"e_1_3_1_5_2","doi-asserted-by":"crossref","unstructured":"Boyd D. and Sarathy J. Differential perspectives: Epistemic disconnects surrounding the U.S. Census Bureau\u2019s use of differential privacy. Harvard Data Science Rev. Special Issue 2 (2022).","DOI":"10.1162\/99608f92.66882f0e"},{"key":"e_1_3_1_6_2","doi-asserted-by":"crossref","unstructured":"Bun M. and Steinke T.\u00a0Concentrated differential privacy: Simplifications extensions and lower bounds.\u00a0In Proceedings of the Theory of Cryptography: 14th Intern. Conf. TCC 2016-B Part I Springer (2016) 635\u2013658.","DOI":"10.1007\/978-3-662-53641-4_24"},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.14778\/3476249.3476272"},{"key":"e_1_3_1_8_2","first-page":"156","article-title":"Three dimensions of reproducibility in natural language processing","author":"Cohen K.B.","year":"2018","unstructured":"Cohen, K.B. et al. Three dimensions of reproducibility in natural language processing. In Proceedings of the Intern. Conf. on Language Resources & Evaluation\u00a0(2018), 156.","journal-title":"Proceedings of the Intern. Conf. on Language Resources & Evaluation"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.3886\/ICPSR36423.v1"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i01.5402"},{"key":"e_1_3_1_11_2","first-page":"202","volume-title":"Proceedings of the 22nd ACM SIGACT-SIGMOD-SIGART Symp. on Principles of Database Systems","author":"Dinur I.","year":"2003","unstructured":"Dinur, I. and Nissim, K.\u00a0Revealing information while preserving privacy. In\u00a0Proceedings of the 22nd ACM SIGACT-SIGMOD-SIGART Symp. on Principles of Database Systems, F.\u00a0Neven, C.\u00a0Beeri, and T.\u00a0Milo\u00a0(eds.),\u00a0ACM\u00a0(2003), 202\u2013210."},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1561\/0400000042"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.67995"},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11121-018-0908-3"},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1002\/ajcp.12233"},{"key":"e_1_3_1_16_2","unstructured":"Hardt M. Ligett K. and McSherry F. A simple and practical algorithm for differentially private data release. arXiv (2010)."},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","unstructured":"Harris K.M. and Udry J.R. National Longitudinal Study of Adolescent to Adult Health (Add Health) 1994-2018. Technical Report ICPSR21600.v25 Carolina Population Center University of North Carolina-Chapel Hill\u00a0(2022); DOI:\u00a010.3886\/ICPSR21600.v25.","DOI":"10.3886\/ICPSR21600.v25"},{"key":"e_1_3_1_18_2","doi-asserted-by":"crossref","unstructured":"Hay M. et al. Principled evaluation of differentially private algorithms using dpbench. In Proceedings of the 2016 Intern. Conf. on Management of Data\u00a0(2016) 139\u2013154.","DOI":"10.1145\/2882903.2882931"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","unstructured":"House J.S. Burgard \u00a0S.A. Hicken \u00a0M.T. and\u00a0Lantz \u00a0P.M.\u00a0Americans\u2019 Changing Lives: Waves I II III IV and V 1986 1989 1994 2002 and 2011. Technical Report ICPSR04690.v9\u00a0(2018); DOI:\u00a010.3886\/ICPSR04690.v9.","DOI":"10.3886\/ICPSR04690.v9"},{"key":"e_1_3_1_20_2","article-title":"High school football and risk for depression and suicidality in adulthood: Findings from a national longitudinal study","volume":"12","author":"Iverson G. L.","year":"2021","unstructured":"Iverson, G. L. and Terry, D.P. High school football and risk for depression and suicidality in adulthood: Findings from a national longitudinal study. Frontiers in Neurology\u00a012\u00a0(2021).","journal-title":"Frontiers in Neurology"},{"key":"e_1_3_1_21_2","unstructured":"Jeong H. et al. Who gets the benefit of the doubt? Racial bias in machine learning algorithms applied to secondary school math education. In Proceedings of the 35th Conf. on Neural Information Processing Systems Workshop on Math AI for Education\u00a0(2021)."},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1126\/sciadv.abk3283"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.adolescence.2021.01.008"},{"key":"e_1_3_1_24_2","first-page":"690","article-title":"Iterative methods for private synthetic data: Unifying framework and new methods","volume":"34","author":"Liu T.","year":"2021","unstructured":"Liu, T., Vietri, G., and Wu, S.Z. Iterative methods for private synthetic data: Unifying framework and new methods. Advances in Neural Information Processing Systems\u00a034\u00a0(2021), 690\u2013702.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_25_2","article-title":"Optimizing error of high-dimensional statistical queries under differential privacy","author":"McKenna R.","year":"2018","unstructured":"McKenna, R., Miklau, G., Hay, M., and Machanavajjhala, A. Optimizing error of high-dimensional statistical queries under differential privacy. arXiv\u00a0(2018).","journal-title":"arXiv"},{"key":"e_1_3_1_26_2","doi-asserted-by":"crossref","unstructured":"McKenna R. Miklau G. and Sheldon D. Winning the NIST contest: A scalable and general approach to differentially private synthetic data. arXiv preprint arXiv:2108.04978 2021.","DOI":"10.29012\/jpc.778"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.14778\/3551793.3551817"},{"key":"e_1_3_1_28_2","unstructured":"National Academies of Sciences Engineering and Medicine and others.\u00a0Reproducibility and Replicability in Science. (2019)."},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.1177\/0265407518817400"},{"key":"e_1_3_1_30_2","unstructured":"Rosenblatt L. Allen J. and Stoyanovich J. Spending privacy budget fairly and wisely. arXiv\u00a0(2022)."},{"key":"e_1_3_1_31_2","doi-asserted-by":"crossref","unstructured":"Rosenblatt L. et al. Epistemic parity: Reproducibility as an evaluation metric for differential privacy. In\u00a0Proceedings of the VLDB Endowment\u00a016 11\u00a0(2023) 3178\u20133191.","DOI":"10.14778\/3611479.3611517"},{"key":"e_1_3_1_32_2","unstructured":"Rosenblatt L. Howe B. and Stoyanovich J. Are data experts buying into differentially private synthetic data? Gathering community perspectives. arXiv (2024)."},{"key":"e_1_3_1_33_2","unstructured":"Rosenblatt L. et al. Differentially private synthetic data: Applied evaluations and enhancements. arXiv (2020)."},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.86.3.638"},{"key":"e_1_3_1_35_2","first-page":"403","article-title":"Differential privacy and census data: Implications for social and economic research","volume":"109","author":"Ruggles S.","year":"2019","unstructured":"Ruggles, S., Fitch, C., Magnuson, D., and Schroeder, J. Differential privacy and census data: Implications for social and economic research. In American Economic Association Papers and Proceedings\u00a0109\u00a0(2019), 403\u201308.","journal-title":"American Economic Association Papers and Proceedings"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.3102\/0013189X18787818"},{"key":"e_1_3_1_37_2","doi-asserted-by":"crossref","unstructured":"Takagi S. Takahashi T. Cao Y. and Yoshikawa M.\u00a0P3gm: Private high-dimensional data release via privacy preserving phased generative model. In Proceedings of the\u00a02021 IEEE 37th Intern. Conf. on Data Engineering IEEE\u00a0(2021) 169\u2013180.","DOI":"10.1109\/ICDE51399.2021.00022"},{"key":"e_1_3_1_38_2","unstructured":"Tao Y. et al. Benchmarking differentially private synthetic data generation algorithms. arXiv (2021)."},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","unstructured":"United States Department of Health and Human Services. National Survey on Drug Use and Health 2014. Technical Report ICPSR36361.v1 (2016); DOI:10.3886\/ICPSR36361.v1.","DOI":"10.3886\/ICPSR36361.v1"},{"key":"e_1_3_1_40_2","article-title":"Modeling tabular data using conditional GAN","volume":"32","author":"Xu L.","year":"2019","unstructured":"Xu, L., Skoularidou, M., Cuesta-Infante, A., and Veeramachaneni, K. Modeling tabular data using conditional GAN. Advances in Neural Information Processing Systems\u00a032\u00a0(2019).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3134428"}],"container-title":["Communications of the ACM"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3807967","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T21:13:27Z","timestamp":1782854007000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3807967"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,30]]},"references-count":40,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2026,7,1]]}},"alternative-id":["10.1145\/3807967"],"URL":"https:\/\/doi.org\/10.1145\/3807967","relation":{},"ISSN":["0001-0782","1557-7317"],"issn-type":[{"value":"0001-0782","type":"print"},{"value":"1557-7317","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,30]]},"assertion":[{"value":"2026-06-30","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}