{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,16]],"date-time":"2026-09-16T21:39:04Z","timestamp":1789594744153,"version":"build-2803163510"},"reference-count":36,"publisher":"MDPI AG","issue":"14","license":[{"start":{"date-parts":[[2023,7,19]],"date-time":"2023-07-19T00:00:00Z","timestamp":1689724800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001858","name":"Area of Advance Transport and Chalmers AI Research Centre (CHAIR)","doi-asserted-by":"publisher","award":["2019-05884"],"award-info":[{"award-number":["2019-05884"]}],"id":[{"id":"10.13039\/501100001858","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001858","name":"Area of Advance Transport and Chalmers AI Research Centre (CHAIR)","doi-asserted-by":"publisher","award":["2018-05973"],"award-info":[{"award-number":["2018-05973"]}],"id":[{"id":"10.13039\/501100001858","id-type":"DOI","asserted-by":"publisher"}]},{"name":"AstraZeneca AB","award":["2019-05884"],"award-info":[{"award-number":["2019-05884"]}]},{"name":"AstraZeneca AB","award":["2018-05973"],"award-info":[{"award-number":["2018-05973"]}]},{"name":"Swedish National Infrastructure for Computing (SNIC) at C3SE","award":["2019-05884"],"award-info":[{"award-number":["2019-05884"]}]},{"name":"Swedish National Infrastructure for Computing (SNIC) at C3SE","award":["2018-05973"],"award-info":[{"award-number":["2018-05973"]}]},{"name":"Swedish Research Council","award":["2019-05884"],"award-info":[{"award-number":["2019-05884"]}]},{"name":"Swedish Research Council","award":["2018-05973"],"award-info":[{"award-number":["2018-05973"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Differential privacy (DP) defines privacy protection by promising quantified indistinguishability between individuals who consent to share their privacy-sensitive information and those who do not. DP aims to deliver this promise by including well-crafted elements of random noise in the published data, and thus there is an inherent tradeoff between the degree of privacy protection and the ability to utilize the protected data. Currently, several open-source tools have been proposed for DP provision. To the best of our knowledge, there is no comprehensive study for comparing these open-source tools with respect to their ability to balance DP\u2019s inherent tradeoff as well as the use of system resources. This work proposes an open-source evaluation framework for privacy protection solutions and offers evaluation for OpenDP Smartnoise, Google DP, PyTorch Opacus, Tensorflow Privacy, and Diffprivlib. In addition to studying their ability to balance the above tradeoff, we consider discrete and continuous attributes by quantifying their performance under different data sizes. Our results reveal several patterns that developers should have in mind when selecting tools under different application needs and criteria. This evaluation survey can be the basis for an improved selection of open-source DP tools and quicker adaptation of DP.<\/jats:p>","DOI":"10.3390\/s23146509","type":"journal-article","created":{"date-parts":[[2023,7,19]],"date-time":"2023-07-19T01:02:23Z","timestamp":1689728543000},"page":"6509","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Evaluation of Open-Source Tools for Differential Privacy"],"prefix":"10.3390","volume":"23","author":[{"given":"Shiliang","family":"Zhang","sequence":"first","affiliation":[{"name":"Computer Science and Engineering, Chalmers University of Technology, SE-41296 Gothenburg, Sweden"},{"name":"Department of Informatics, University of Oslo, NO-0373 Oslo, Norway"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anton","family":"Hagermalm","sequence":"additional","affiliation":[{"name":"Computer Science and Engineering, Chalmers University of Technology, SE-41296 Gothenburg, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sanjin","family":"Slavnic","sequence":"additional","affiliation":[{"name":"Computer Science and Engineering, Chalmers University of Technology, SE-41296 Gothenburg, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3258-3696","authenticated-orcid":false,"given":"Elad Michael","family":"Schiller","sequence":"additional","affiliation":[{"name":"Computer Science and Engineering, Chalmers University of Technology, SE-41296 Gothenburg, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3383-9617","authenticated-orcid":false,"given":"Magnus","family":"Almgren","sequence":"additional","affiliation":[{"name":"Computer Science and Engineering, Chalmers University of Technology, SE-41296 Gothenburg, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,7,19]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Scherer, J., and Kiparski, G. (2018). Buchbesprechungen. Feiler, Lukas\/Forg\u00f3, Nikolaus\/Weigl, Michaela: The EU General Data Protection Regulation (GDPR): A Commentary, German Law Publishers.","DOI":"10.9785\/cr-2018-340626"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1927495:1","DOI":"10.1155\/2019\/1927495","article-title":"Healthcare Data Security Technology: HIPAA Compliance","volume":"2019","author":"Mbonihankuye","year":"2019","journal-title":"Wirel. Commun. Mob. Comput."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Goldman, E. (2020). An Introduction to the California Consumer Privacy Act (CCPA), Santa Clara University Legal Studies Research Paper.","DOI":"10.4337\/9781788119924.00025"},{"key":"ref_4","first-page":"1","article-title":"Differential Privacy: A Survey of Results","volume":"Volume 4978","author":"Agrawal","year":"2008","journal-title":"Proceedings of the Theory and Applications of Models of Computation, 5th International Conference, TAMC 2008, Xi\u2019an, China, 25\u201329 April 2008"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"17","DOI":"10.29012\/jpc.v7i3.405","article-title":"Calibrating Noise to Sensitivity in Private Data Analysis","volume":"7","author":"Dwork","year":"2016","journal-title":"J. Priv. Confidentiality"},{"key":"ref_6","unstructured":"(2023, July 07). Harvard University Privacy Tools Project. Available online: https:\/\/privacytools.seas.harvard.edu\/."},{"key":"ref_7","unstructured":"Hsu, J., Gaboardi, M., Haeberlen, A., Khanna, S., Narayan, A., Pierce, B.C., and Roth, A. (2014). Proceedings of the IEEE 27th Computer Security Foundations Symposium, CSF 2014, Vienna, Austria, 19\u201322 July 2014, IEEE Computer Society."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1145\/1810891.1810916","article-title":"Privacy integrated queries: An extensible platform for privacy-preserving data analysis","volume":"53","author":"McSherry","year":"2010","journal-title":"Commun. ACM"},{"key":"ref_9","unstructured":"Zhang, D., McKenna, R., Kotsogiannis, I., Hay, M., Machanavajjhala, A., and Miklau, G. (2018). Proceedings of the International Conference on Management of Data, SIGMOD, Houston, TX, USA, 10\u201315 June 2018, ACM."},{"key":"ref_10","unstructured":"Kotsogiannis, I., Tao, Y., Machanavajjhala, A., Miklau, G., and Hay, M. (2019, January 13\u201316). Architecting a Differentially Private SQL Engine. Proceedings of the CIDR, Asilomar, CA, USA. Available online: www.cidrdb.org."},{"key":"ref_11","unstructured":"Haeberlen, A., Pierce, B.C., and Narayan, A. (2011). Proceedings of the 20th USENIX Security Symposium, San Francisco, CA, USA, 8\u201312 August 2011, USENIX Association."},{"key":"ref_12","unstructured":"Zhang, D., and Kifer, D. (2017). Proceedings of the POPL, Paris, France, 15\u201321 January 2017, ACM."},{"key":"ref_13","unstructured":"(2021, June 04). Miguel Guevara Enabling Developers and Organizations to Use Differential Privacy. Available online: https:\/\/developers.googleblog.com\/2019\/09\/enabling-developers-and-organizations.html."},{"key":"ref_14","unstructured":"(2023, July 07). Openminded. Available online: https:\/\/www.openminded.org\/."},{"key":"ref_15","unstructured":"Subramani, P., Vadivelu, N., and Kamath, G. (2020). Enabling Fast Differentially Private SGD via Just-in-Time Compilation and Vectorization. arXiv."},{"key":"ref_16","unstructured":"Abadi, M., Chu, A., Goodfellow, I.J., McMahan, H.B., Mironov, I., Talwar, K., and Zhang, L. (2016). Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, 24\u201328 October 2016, ACM."},{"key":"ref_17","unstructured":"Holohan, N., Braghin, S., Aonghusa, P.M., and Levacher, K. (2019). Diffprivlib: The IBM Differential Privacy Library. arXiv."},{"key":"ref_18","unstructured":"(2021, June 02). Diffprivlib. Available online: https:\/\/github.com\/IBM\/differential-privacy-library."},{"key":"ref_19","unstructured":"Johnson, N.M., Near, J.P., Hellerstein, J.M., and Song, D. (2018). Chorus: Differential Privacy via Query Rewriting. arXiv."},{"key":"ref_20","unstructured":"Johnson, N.M., Near, J.P., Hellerstein, J.M., and Song, D. (2020). Proceedings of the IEEE European Symposium on Security and Privacy, EuroS&P 2020, Genoa, Italy, 7\u201311 September 2020, IEEE."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"230","DOI":"10.2478\/popets-2020-0025","article-title":"Differentially Private SQL with Bounded User Contribution","volume":"2020","author":"Wilson","year":"2020","journal-title":"Proc. Priv. Enhancing Technol."},{"key":"ref_22","unstructured":"Haney, S., Desfontaines, D., Hartman, L., Shrestha, R., and Hay, M. (2022). Precision-based attacks and interval refining: How to break, then fix, differential privacy on finite computers. arXiv."},{"key":"ref_23","unstructured":"Long, S.K. (2019). Massachusetts Health Reform Survey, 2018, Inter-University Consortium for Political and Social Research."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"884","DOI":"10.1109\/TBME.2009.2036000","article-title":"Accurate Telemonitoring of Parkinson\u2019s Disease Progression by Noninvasive Speech Tests","volume":"57","author":"Tsanas","year":"2010","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1145\/2723872.2723882","article-title":"An introduction to Docker for reproducible research","volume":"49","author":"Boettiger","year":"2015","journal-title":"ACM SIGOPS Oper. Syst. Rev."},{"key":"ref_26","unstructured":"Kopp, A. (Microsoft Azure White Papers, 2021). Microsoft SmartNoise Differential Privacy Machine Learning Case Studies, Microsoft Azure White Papers."},{"key":"ref_27","unstructured":"Rosenblatt, L., Liu, X., Pouyanfar, S., de Leon, E., Desai, A., and Allen, J. (2020). Differentially Private Synthetic Data: Applied Evaluations and Enhancements. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"526","DOI":"10.1145\/3187009.3177733","article-title":"Towards Practical Differential Privacy for SQL Queries","volume":"11","author":"Johnson","year":"2018","journal-title":"Proc. VLDB Endow."},{"key":"ref_29","unstructured":"Garrido, G.M., Near, J.P., Muhammad, A., He, W., Matzutt, R., and Matthes, F. (2021). Do I Get the Privacy I Need? Benchmarking Utility in Differential Privacy Libraries. arXiv."},{"key":"ref_30","unstructured":"\u00d6zcan, F., Koutrika, G., and Madden, S. (July, January 26). Principled Evaluation of Differentially Private Algorithms using DPBench. Proceedings of the International Conference on Management of Data, SIGMOD, San Francisco, CA, USA."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Bu, Z., Gopi, S., Kulkarni, J., Lee, Y.T., Shen, J.H., and Tantipongpipat, U. (2021). Fast and Memory Efficient Differentially Private-SGD via JL Projections. arXiv.","DOI":"10.29012\/jpc.780"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"797","DOI":"10.1007\/s00778-013-0309-y","article-title":"Differentially private histogram publication","volume":"22","author":"Xu","year":"2013","journal-title":"VLDB J."},{"key":"ref_33","unstructured":"Heninger, N., and Traynor, P. (2019). Proceedings of the 28th USENIX Security Symposium, USENIX Security 2019, Santa Clara, CA, USA, 14\u201316 August 2019, USENIX Association."},{"key":"ref_34","unstructured":"Mironov, I. (2017). Proceedings of the 30th IEEE Computer Security Foundations Symposium, CSF 2017, Santa Barbara, CA, USA, 21\u201325 August 2017, IEEE Computer Society."},{"key":"ref_35","unstructured":"Tram\u00e8r, F., and Boneh, D. (2021). Differentially Private Learning Needs Better Features (or Much More Data). arXiv."},{"key":"ref_36","unstructured":"Zhang, S., Hagermalm, A., Slavnic, S., Schiller, E.M., and Almgren, M. (2022). Evaluation of Open-source Tools for Differential Privacy. arXiv."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/14\/6509\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:14:28Z","timestamp":1760127268000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/14\/6509"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,19]]},"references-count":36,"journal-issue":{"issue":"14","published-online":{"date-parts":[[2023,7]]}},"alternative-id":["s23146509"],"URL":"https:\/\/doi.org\/10.3390\/s23146509","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,7,19]]}}}