{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,9]],"date-time":"2025-12-09T08:26:41Z","timestamp":1765268801455},"reference-count":44,"publisher":"Privacy Enhancing Technologies Symposium Advisory Board","issue":"1","license":[{"start":{"date-parts":[[2021,11,20]],"date-time":"2021-11-20T00:00:00Z","timestamp":1637366400000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc-nd\/3.0"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,1,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>A membership inference attack (MIA) against a machine-learning model enables an attacker to determine whether a given data record was part of the model\u2019s training data or not. In this paper, we provide an in-depth study of the phenomenon of <jats:italic>disparate vulnerability<\/jats:italic> against MIAs: unequal success rate of MIAs against different population subgroups. We first establish necessary and sufficient conditions for MIAs to be prevented, both on average and for population subgroups, using a notion of distributional generalization. Second, we derive connections of disparate vulnerability to algorithmic fairness and to differential privacy. We show that fairness can only prevent disparate vulnerability against limited classes of adversaries. Differential privacy bounds disparate vulnerability but can significantly reduce the accuracy of the model. We show that estimating disparate vulnerability by na\u00efvely applying existing attacks can lead to overestimation. We then establish which attacks are suitable for estimating disparate vulnerability, and provide a statistical framework for doing so reliably. We conduct experiments on synthetic and real-world data finding significant evidence of disparate vulnerability in realistic settings.<\/jats:p>","DOI":"10.2478\/popets-2022-0023","type":"journal-article","created":{"date-parts":[[2021,11,21]],"date-time":"2021-11-21T02:44:48Z","timestamp":1637462688000},"page":"460-480","source":"Crossref","is-referenced-by-count":16,"title":["Disparate Vulnerability to Membership Inference Attacks"],"prefix":"10.56553","volume":"2022","author":[{"given":"Bogdan","family":"Kulynych","sequence":"first","affiliation":[{"name":"EPFL"}]},{"given":"Mohammad","family":"Yaghini","sequence":"additional","affiliation":[{"name":"University of Toronto , Vector Institute"}]},{"given":"Giovanni","family":"Cherubin","sequence":"additional","affiliation":[{"name":"Alan Turing Institute"}]},{"given":"Michael","family":"Veale","sequence":"additional","affiliation":[{"name":"University College London"}]},{"given":"Carmela","family":"Troncoso","sequence":"additional","affiliation":[{"name":"EPFL"}]}],"member":"35752","published-online":{"date-parts":[[2021,11,20]]},"reference":[{"key":"2022062314355760507_j_popets-2022-0023_ref_001","unstructured":"[1] Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. Differential privacy has disparate impact on model accuracy. In Annual Conference on Neural Information Processing Systems, NeurIPS, 2019."},{"key":"2022062314355760507_j_popets-2022-0023_ref_002","doi-asserted-by":"crossref","unstructured":"[2] Solon Barocas and Andrew D Selbst. Big data\u2019s disparate impact. Calif. L. Rev., 2016.10.2139\/ssrn.2477899","DOI":"10.2139\/ssrn.2477899"},{"key":"2022062314355760507_j_popets-2022-0023_ref_003","unstructured":"[3] Arindrajit Basu, Elonnai Hickok, and Aditya Singh Chawala. The Localisation Gambit: Unpacking Policy Measures for Sovereign Control of Data in India. Centre for Internet and Society, India, 2019."},{"key":"2022062314355760507_j_popets-2022-0023_ref_004","doi-asserted-by":"crossref","unstructured":"[4] Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth. Fairness in criminal justice risk assessments: The state of the art. Sociological Methods & Research, 2018.10.1177\/0049124118782533","DOI":"10.1177\/0049124118782533"},{"key":"2022062314355760507_j_popets-2022-0023_ref_005","unstructured":"[5] Sarah Bird, Miro Dud\u00edk, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, and Kathleen Walker. Fairlearn: A toolkit for assessing and improving fairness in AI. Technical Report MSR-TR-2020-32, Microsoft, May 2020. URL https:\/\/www.microsoft.com\/en-us\/research\/publication\/fairlearn-atoolkit-for-assessing-and-improving-fairness-in-ai\/."},{"key":"2022062314355760507_j_popets-2022-0023_ref_006","doi-asserted-by":"crossref","unstructured":"[6] Hongyan Chang and Reza Shokri. On the privacy risks of algorithmic fairness. IEEE European Symposium on Security and Privacy, EuroS&P, 2021.10.1109\/EuroSP51992.2021.00028","DOI":"10.1109\/EuroSP51992.2021.00028"},{"key":"2022062314355760507_j_popets-2022-0023_ref_007","unstructured":"[7] Konstantinos Chatzikokolakis, Giovanni Cherubin, Catuscia Palamidessi, and Carmela Troncoso. The Bayes security measure. arXiv preprint arXiv:2011.03396, 2020."},{"key":"2022062314355760507_j_popets-2022-0023_ref_008","unstructured":"[8] Kamalika Chaudhuri, Claire Monteleoni, and Anand D. Sarwate. Differentially private empirical risk minimization. J. Mach. Learn. Res., 2011."},{"key":"2022062314355760507_j_popets-2022-0023_ref_009","doi-asserted-by":"crossref","unstructured":"[9] Giovanni Cherubin, Konstantinos Chatzikokolakis, and Catuscia Palamidessi. F-BLEAU: Fast black-box leakage estimation. In IEEE Symposium on Security and Privacy, S&P, 2019.10.1109\/SP.2019.00073","DOI":"10.1109\/SP.2019.00073"},{"key":"2022062314355760507_j_popets-2022-0023_ref_010","doi-asserted-by":"crossref","unstructured":"[10] Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data, 2017.10.1089\/big.2016.004728632438","DOI":"10.1089\/big.2016.0047"},{"key":"2022062314355760507_j_popets-2022-0023_ref_011","unstructured":"[11] Alexandra Chouldechova and Aaron Roth. The frontiers of fairness in machine learning. arXiv preprint arXiv:1810.08810, 2018."},{"key":"2022062314355760507_j_popets-2022-0023_ref_012","unstructured":"[12] Luc Devroye, L\u00e1szl\u00f3 Gy\u00f6rfi, and G\u00e1bor Lugosi. A probabilistic theory of pattern recognition, volume 31. Springer Science & Business Media, 2013."},{"key":"2022062314355760507_j_popets-2022-0023_ref_013","doi-asserted-by":"crossref","unstructured":"[13] Cynthia Dwork. Differential Privacy. Springer US, 2011.10.4016\/26354.01","DOI":"10.4016\/26354.01"},{"key":"2022062314355760507_j_popets-2022-0023_ref_014","doi-asserted-by":"crossref","unstructured":"[14] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Rein-gold, and Richard S. Zemel. Fairness through awareness. In Innovations in Theoretical Computer Science, 2012.10.1145\/2090236.2090255","DOI":"10.1145\/2090236.2090255"},{"key":"2022062314355760507_j_popets-2022-0023_ref_015","unstructured":"[15] Michael D. Ekstrand, Rezvan Joshaghani, and Hoda Mehrpouyan. Privacy for all: Ensuring fair and equitable privacy protections. In Conference on Fairness, Accountability and Transparency, FAT, 2018."},{"key":"2022062314355760507_j_popets-2022-0023_ref_016","unstructured":"[16] Farhad Farokhi and Mohamed Ali Kaafar. Modelling and quantifying membership information leakage in machine learning. arXiv preprint arXiv:2001.10648, 2020."},{"key":"2022062314355760507_j_popets-2022-0023_ref_017","unstructured":"[17] Sorelle A Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian. On the (im) possibility of fairness. arXiv preprint arXiv:1609.07236, 2016."},{"key":"2022062314355760507_j_popets-2022-0023_ref_018","unstructured":"[18] Graham Greenleaf and Bertil Cottier. 2020 ends a decade of 62 new data privacy laws. Privacy Laws & Business International Report, 2020."},{"key":"2022062314355760507_j_popets-2022-0023_ref_019","unstructured":"[19] Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. In NIPS, 2016."},{"key":"2022062314355760507_j_popets-2022-0023_ref_020","unstructured":"[20] Naoise Holohan, Stefano Braghin, P\u00f3l Mac Aonghusa, and Killian Levacher. Diffprivlib: The IBM differential privacy library. arXiv preprint arXiv:1907.02444, 2019."},{"key":"2022062314355760507_j_popets-2022-0023_ref_021","unstructured":"[21] Thomas Humphries, Matthew Rafuse, Lindsey Tulloch, Simon Oya, Ian Goldberg, Urs Hengartner, and Florian Kerschbaum. Differentially private learning does not bound membership inference. arXiv preprint arXiv:2010.12112, 2020."},{"key":"2022062314355760507_j_popets-2022-0023_ref_022","doi-asserted-by":"crossref","unstructured":"[22] Bargav Jayaraman, Lingxiao Wang, David Evans, and Quanquan Gu. Revisiting membership inference under realistic assumptions. Proceedings on Privacy Enhancing Technologies, 2021.10.2478\/popets-2021-0031","DOI":"10.2478\/popets-2021-0031"},{"key":"2022062314355760507_j_popets-2022-0023_ref_023","doi-asserted-by":"crossref","unstructured":"[23] Michael J. Kearns, Yishay Mansour, Dana Ron, Ronitt Rubinfeld, Robert E. Schapire, and Linda Sellie. On the learnability of discrete distributions. In ACM Symposium on Theory of Computing, 1994.10.1145\/195058.195155","DOI":"10.1145\/195058.195155"},{"key":"2022062314355760507_j_popets-2022-0023_ref_024","doi-asserted-by":"crossref","unstructured":"[24] Amir E Khandani, Adlar J Kim, and Andrew W Lo. Consumer credit-risk models via machine-learning algorithms. Journal of Banking & Finance, 2010.10.2139\/ssrn.1568864","DOI":"10.2139\/ssrn.1568864"},{"key":"2022062314355760507_j_popets-2022-0023_ref_025","unstructured":"[25] Ron Kohavi. Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid. In International Conference on Knowledge Discovery and Data Mining, KDD, 1996."},{"key":"2022062314355760507_j_popets-2022-0023_ref_026","unstructured":"[26] Klas Leino and Matt Fredrikson. Stolen memories: Leveraging model memorization for calibrated white-box membership inference. In Srdjan Capkun and Franziska Roesner, editors, USENIX Security Symposium, 2020."},{"key":"2022062314355760507_j_popets-2022-0023_ref_027","unstructured":"[27] Jiacheng Li, Ninghui Li, and Bruno Ribeiro. Membership inference attacks and defenses in classification models. In CODASPY, 2021."},{"key":"2022062314355760507_j_popets-2022-0023_ref_028","unstructured":"[28] Zachary C. Lipton, Julian McAuley, and Alexandra Chouldechova. Does mitigating ML\u2019s impact disparity require treatment disparity? In Annual Conference on Neural Information Processing Systems,NeurIPS, 2018."},{"key":"2022062314355760507_j_popets-2022-0023_ref_029","doi-asserted-by":"crossref","unstructured":"[29] Yunhui Long, Lei Wang, Diyue Bu, Vincent Bindschaedler, Xiaofeng Wang, Haixu Tang, Carl A Gunter, and Kai Chen. A pragmatic approach to membership inferences on machine learning models. In IEEE European Symposium on Security and Privacy, EuroS&P, 2020.10.1109\/EuroSP48549.2020.00040","DOI":"10.1109\/EuroSP48549.2020.00040"},{"key":"2022062314355760507_j_popets-2022-0023_ref_030","unstructured":"[30] Kristian Lum and William Isaac. To predict and serve? Significance, 2016."},{"key":"2022062314355760507_j_popets-2022-0023_ref_031","unstructured":"[31] Preetum Nakkiran and Yamini Bansal. Distributional generalization: A new kind of generalization. arXiv preprint arXiv:2009.08092, 2020."},{"key":"2022062314355760507_j_popets-2022-0023_ref_032","unstructured":"[32] Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning: Stand-alone and federated learning under passive and active white-box inference attacks. In IEEE Symposium on Security and Privacy, S&P, 2018."},{"key":"2022062314355760507_j_popets-2022-0023_ref_033","doi-asserted-by":"crossref","unstructured":"[33] Ziad Obermeyer and Ezekiel J Emanuel. Predicting the future\u2014big data, machine learning, and clinical medicine. The New England journal of medicine, 2016.10.1056\/NEJMp1606181507053227682033","DOI":"10.1056\/NEJMp1606181"},{"key":"2022062314355760507_j_popets-2022-0023_ref_034","unstructured":"[34] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 2011."},{"key":"2022062314355760507_j_popets-2022-0023_ref_035","doi-asserted-by":"crossref","unstructured":"[35] David Pujol, Ryan McKenna, Satya Kuppam, Michael Hay, Ashwin Machanavajjhala, and Gerome Miklau. Fair decision making using privacy-protected data. In Conference on Fairness, Accountability, and Transparency, FAT*, 2020.10.1145\/3351095.3372872","DOI":"10.1145\/3351095.3372872"},{"key":"2022062314355760507_j_popets-2022-0023_ref_036","unstructured":"[36] Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Herv\u00e9 J\u00e9gou. White-box vs black-box: Bayes optimal strategies for membership inference. In International Conference on Machine Learning, ICML, 2019."},{"key":"2022062314355760507_j_popets-2022-0023_ref_037","doi-asserted-by":"crossref","unstructured":"[37] Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. ML-leaks: Model and data independent membership inference attacks and defenses on machine learning models. In 26th Annual Network and Distributed System Security Symposium, NDSS, 2019.10.14722\/ndss.2019.23119","DOI":"10.14722\/ndss.2019.23119"},{"key":"2022062314355760507_j_popets-2022-0023_ref_038","unstructured":"[38] Howard J Seltman. Experimental design and analysis. 2012."},{"key":"2022062314355760507_j_popets-2022-0023_ref_039","doi-asserted-by":"crossref","unstructured":"[39] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In IEEE Symposium on Security and Privacy, S&P, 2017.10.1109\/SP.2017.41","DOI":"10.1109\/SP.2017.41"},{"key":"2022062314355760507_j_popets-2022-0023_ref_040","unstructured":"[40] Reza Shokri, Martin Strobel, and Yair Zick. On the privacy risks of model explanations. arXiv preprint arXiv:1907.00164, 2019."},{"key":"2022062314355760507_j_popets-2022-0023_ref_041","unstructured":"[41] Liwei Song and Prateek Mittal. Systematic evaluation of privacy risks of machine learning models. In USENIX Security Symposium, 2021."},{"key":"2022062314355760507_j_popets-2022-0023_ref_042","doi-asserted-by":"crossref","unstructured":"[42] Michael Veale, Reuben Binns, and Lilian Edwards. Algorithms that remember: model inversion attacks and data protection law. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 2018.","DOI":"10.31228\/osf.io\/4bmfv"},{"key":"2022062314355760507_j_popets-2022-0023_ref_043","doi-asserted-by":"crossref","unstructured":"[43] Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In IEEE Computer Security Foundations Symposium, CSF, 2018.10.1109\/CSF.2018.00027","DOI":"10.1109\/CSF.2018.00027"},{"key":"2022062314355760507_j_popets-2022-0023_ref_044","unstructured":"[44] Han Zhao and Geoffrey J. Gordon. Inherent tradeoffs in learning fair representations. In Annual Conference on Neural Information Processing Systems, NeurIPS, 2019."}],"container-title":["Proceedings on Privacy Enhancing Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.sciendo.com\/pdf\/10.2478\/popets-2022-0023","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,7,20]],"date-time":"2022-07-20T16:31:56Z","timestamp":1658334716000},"score":1,"resource":{"primary":{"URL":"https:\/\/petsymposium.org\/popets\/2022\/popets-2022-0023.php"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,20]]},"references-count":44,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2021,11,20]]},"published-print":{"date-parts":[[2022,1,1]]}},"alternative-id":["10.2478\/popets-2022-0023"],"URL":"https:\/\/doi.org\/10.2478\/popets-2022-0023","relation":{},"ISSN":["2299-0984"],"issn-type":[{"value":"2299-0984","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,11,20]]}}}