{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,4]],"date-time":"2026-07-04T17:03:15Z","timestamp":1783184595536,"version":"3.54.6"},"reference-count":33,"publisher":"Association for Computing Machinery (ACM)","issue":"11","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2023,7]]},"abstract":"<jats:p>\n            We address two major obstacles to practical deployment of AI-based models on distributed private data. Whether a model was trained by a federation of cooperating clients or trained centrally, (1) the output scores must be calibrated, and (2) performance metrics must be evaluated --- all without assembling labels in one place. In particular, we show how to perform calibration and compute the standard metrics of precision, recall, accuracy and ROC-AUC in the federated setting under three privacy models (\n            <jats:italic>i<\/jats:italic>\n            ) secure aggregation, (\n            <jats:italic>ii<\/jats:italic>\n            ) distributed differential privacy, (\n            <jats:italic>iii<\/jats:italic>\n            ) local differential privacy. Our theorems and experiments clarify tradeoffs between privacy, accuracy, and data efficiency. They also help decide if a given application has sufficient data to support federated calibration and evaluation.\n          <\/jats:p>","DOI":"10.14778\/3611479.3611523","type":"journal-article","created":{"date-parts":[[2023,8,25]],"date-time":"2023-08-25T02:08:08Z","timestamp":1692929288000},"page":"3253-3265","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Federated Calibration and Evaluation of Binary Classifiers"],"prefix":"10.14778","volume":"16","author":[{"given":"Graham","family":"Cormode","sequence":"first","affiliation":[{"name":"Meta"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Igor L.","family":"Markov","sequence":"additional","affiliation":[{"name":"Meta"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,8,24]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"The Skellam Mechanism for Differentially Private Federated Learning. CoRR abs\/2110.04995","author":"Agarwal Naman","year":"2021","unstructured":"Naman Agarwal , Peter Kairouz , and Ziyu Liu . 2021. The Skellam Mechanism for Differentially Private Federated Learning. CoRR abs\/2110.04995 ( 2021 ), 25. arXiv:2110.04995 https:\/\/arxiv.org\/abs\/2110.04995 Naman Agarwal, Peter Kairouz, and Ziyu Liu. 2021. The Skellam Mechanism for Differentially Private Federated Learning. CoRR abs\/2110.04995 (2021), 25. arXiv:2110.04995 https:\/\/arxiv.org\/abs\/2110.04995"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3372297.3417242"},{"key":"e_1_2_1_3_1","doi-asserted-by":"crossref","DOI":"10.1002\/9781119642183","volume-title":"AISTATS (Proc. Machine Learning Research)","volume":"108","author":"Bell James","year":"2020","unstructured":"James Bell , Aur\u00e9lien Bellet , Adri\u00e0 Gasc\u00f3n , and Tejas Kulkarni . 2020 . Private Protocols for U-Statistics in the Local Model and Beyond. In Int'l Conf. Artificial Intelligence and Statistics , AISTATS (Proc. Machine Learning Research) , Vol. 108 . PMLR, 1573--1583. http:\/\/proceedings.mlr.press\/v108\/bell20a.html James Bell, Aur\u00e9lien Bellet, Adri\u00e0 Gasc\u00f3n, and Tejas Kulkarni. 2020. Private Protocols for U-Statistics in the Local Model and Beyond. In Int'l Conf. Artificial Intelligence and Statistics, AISTATS (Proc. Machine Learning Research), Vol. 108. PMLR, 1573--1583. http:\/\/proceedings.mlr.press\/v108\/bell20a.html"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1914598117"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.14778\/3339490.3339496"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2210.12526"},{"key":"e_1_2_1_7_1","unstructured":"Damien Desfontaines. 2021. A list of real-world uses of differential privacy. https:\/\/desfontain.es\/privacy\/real-world-differential-privacy.html.  Damien Desfontaines. 2021. A list of real-world uses of differential privacy. https:\/\/desfontain.es\/privacy\/real-world-differential-privacy.html."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/n19-1423"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/11761679_29"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.29012\/jpc.689"},{"key":"e_1_2_1_11_1","first-page":"3","article-title":"The Algorithmic Foundations of Differential Privacy","volume":"9","author":"Dwork Cynthia","year":"2014","unstructured":"Cynthia Dwork and Aaron Roth . 2014 . The Algorithmic Foundations of Differential Privacy . Foundations and Trends in Theoretical Computer Science 9 , 3 -- 4 (2014), 211--407. http:\/\/dblp.uni-trier.de\/db\/journals\/fttcs\/fttcs9.html#DworkR14 Cynthia Dwork and Aaron Roth. 2014. The Algorithmic Foundations of Differential Privacy. Foundations and Trends in Theoretical Computer Science 9, 3--4 (2014), 211--407. http:\/\/dblp.uni-trier.de\/db\/journals\/fttcs\/fttcs9.html#DworkR14","journal-title":"Foundations and Trends in Theoretical Computer Science"},{"key":"e_1_2_1_12_1","volume-title":"AISTATS (Proc. Machine Learning Research)","volume":"89","author":"Gaboardi Marco","year":"2019","unstructured":"Marco Gaboardi , Ryan Rogers , and Or Sheffet . 2019 . Locally Private Mean Estimation: Z-test and Tight Confidence Intervals. In Int'l Conf. Artificial Intelligence and Statistics , AISTATS (Proc. Machine Learning Research) , Vol. 89 . PMLR, 2545--2554. http:\/\/proceedings.mlr.press\/v89\/gaboardi19a.html Marco Gaboardi, Ryan Rogers, and Or Sheffet. 2019. Locally Private Mean Estimation: Z-test and Tight Confidence Intervals. In Int'l Conf. Artificial Intelligence and Statistics, AISTATS (Proc. Machine Learning Research), Vol. 89. PMLR, 2545--2554. http:\/\/proceedings.mlr.press\/v89\/gaboardi19a.html"},{"key":"e_1_2_1_13_1","volume-title":"On Calibration of Modern Neural Networks. In Proc. 34th International Conference on Machine Learning (Proc. Machine Learning Research)","volume":"70","author":"Guo Chuan","unstructured":"Chuan Guo , Geoff Pleiss , Yu Sun , and Kilian Q. Weinberger . 2017 . On Calibration of Modern Neural Networks. In Proc. 34th International Conference on Machine Learning (Proc. Machine Learning Research) , Vol. 70 . PMLR, 1321--1330. http:\/\/proceedings.mlr.press\/v70\/guo17a.html Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On Calibration of Modern Neural Networks. In Proc. 34th International Conference on Machine Learning (Proc. Machine Learning Research), Vol. 70. PMLR, 1321--1330. http:\/\/proceedings.mlr.press\/v70\/guo17a.html"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1010920819831"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/CSF.2014.35"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1561\/2200000083"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-24861-0_22"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.acra.2013.04.011"},{"key":"e_1_2_1_19_1","volume-title":"AISTATS (Proc. Machine Learning Research)","volume":"54","author":"McMahan Brendan","year":"2017","unstructured":"Brendan McMahan , Eider Moore , Daniel Ramage , Seth Hampson , and Blaise Ag\u00fcera y Arcas . 2017 . Communication-Efficient Learning of Deep Networks from Decentralized Data. In Int'l Conf. Artificial Intelligence and Statistics , AISTATS (Proc. Machine Learning Research) , Vol. 54 . PMLR, 1273--1282. http:\/\/proceedings.mlr.press\/v54\/mcmahan17a.html Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Ag\u00fcera y Arcas. 2017. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Int'l Conf. Artificial Intelligence and Statistics, AISTATS (Proc. Machine Learning Research), Vol. 54. PMLR, 1273--1282. http:\/\/proceedings.mlr.press\/v54\/mcmahan17a.html"},{"key":"e_1_2_1_20_1","volume-title":"Revisiting the Calibration of Modern Neural Networks. CoRR abs\/2106.07998","author":"Minderer Matthias","year":"2021","unstructured":"Matthias Minderer , Josip Djolonga , Rob Romijnders , Frances Hubis , Xiaohua Zhai , Neil Houlsby , Dustin Tran , and Mario Lucic . 2021. Revisiting the Calibration of Modern Neural Networks. CoRR abs\/2106.07998 ( 2021 ), 28. arXiv:2106.07998 https:\/\/arxiv.org\/abs\/2106.07998 Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. 2021. Revisiting the Calibration of Modern Neural Networks. CoRR abs\/2106.07998 (2021), 28. arXiv:2106.07998 https:\/\/arxiv.org\/abs\/2106.07998"},{"key":"e_1_2_1_21_1","volume-title":"Randomized Algorithms","author":"Motwani Rajeev","unstructured":"Rajeev Motwani and Prabhakar Raghavan . 1995. Randomized Algorithms . Cambridge University Press . Rajeev Motwani and Prabhakar Raghavan. 1995. Randomized Algorithms. Cambridge University Press."},{"key":"e_1_2_1_22_1","volume-title":"Obtaining Well Calibrated Probabilities Using Bayesian Binning. In AAAI Conf. Artificial Intelligence. AAAI Press, 2901--2907","author":"Naeini Mahdi Pakdaman","year":"2015","unstructured":"Mahdi Pakdaman Naeini , Gregory F. Cooper , and Milos Hauskrecht . 2015 . Obtaining Well Calibrated Probabilities Using Bayesian Binning. In AAAI Conf. Artificial Intelligence. AAAI Press, 2901--2907 . http:\/\/www.aaai.org\/ocs\/index.php\/AAAI\/AAAI15\/paper\/view\/9667 Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. 2015. Obtaining Well Calibrated Probabilities Using Bayesian Binning. In AAAI Conf. Artificial Intelligence. AAAI Press, 2901--2907. http:\/\/www.aaai.org\/ocs\/index.php\/AAAI\/AAAI15\/paper\/view\/9667"},{"key":"e_1_2_1_23_1","volume-title":"Russell and Peter Norvig","author":"Stuart","year":"2020","unstructured":"Stuart J. Russell and Peter Norvig . 2020 . Artificial Intelligence : A Modern Approach (4th Edition). Pearson . http:\/\/aima.cs.berkeley.edu\/ Stuart J. Russell and Peter Norvig. 2020. Artificial Intelligence: A Modern Approach (4th Edition). Pearson. http:\/\/aima.cs.berkeley.edu\/"},{"key":"e_1_2_1_24_1","unstructured":"Aaron Segal Antonio Marcedone Benjamin Kreuter Daniel Ramage H. Brendan McMahan Karn Seth K. A. Bonawitz Sarvar Patel and Vladimir Ivanov. 2017. Practical Secure Aggregation for Privacy-Preserving Machine Learning. In CCS. ACM 21. https:\/\/eprint.iacr.org\/2017\/281.pdf  Aaron Segal Antonio Marcedone Benjamin Kreuter Daniel Ramage H. Brendan McMahan Karn Seth K. A. Bonawitz Sarvar Patel and Vladimir Ivanov. 2017. Practical Secure Aggregation for Privacy-Preserving Machine Learning. In CCS. ACM 21. https:\/\/eprint.iacr.org\/2017\/281.pdf"},{"key":"e_1_2_1_25_1","volume-title":"Differentially Private Algorithms for Empirical Machine Learning. CoRR abs\/1411.5428","author":"Stoddard Ben","year":"2014","unstructured":"Ben Stoddard , Yan Chen , and Ashwin Machanavajjhala . 2014. Differentially Private Algorithms for Empirical Machine Learning. CoRR abs\/1411.5428 ( 2014 ), 13. arXiv:1411.5428 http:\/\/arxiv.org\/abs\/1411.5428 Ben Stoddard, Yan Chen, and Ashwin Machanavajjhala. 2014. Differentially Private Algorithms for Empirical Machine Learning. CoRR abs\/1411.5428 (2014), 13. arXiv:1411.5428 http:\/\/arxiv.org\/abs\/1411.5428"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2205.12412"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2208.12294"},{"key":"e_1_2_1_28_1","volume-title":"Locally Differentially Private Protocols for Frequency Estimation. In 26th USENIX Security Symposium, USENIX Security. USENIX Association, 729--745","author":"Wang Tianhao","year":"2017","unstructured":"Tianhao Wang , Jeremiah Blocki , Ninghui Li , and Somesh Jha . 2017 . Locally Differentially Private Protocols for Frequency Estimation. In 26th USENIX Security Symposium, USENIX Security. USENIX Association, 729--745 . https:\/\/www.usenix.org\/conference\/usenixsecurity17\/technical-sessions\/presentation\/wang-tianhao Tianhao Wang, Jeremiah Blocki, Ninghui Li, and Somesh Jha. 2017. Locally Differentially Private Protocols for Frequency Estimation. In 26th USENIX Security Symposium, USENIX Security. USENIX Association, 729--745. https:\/\/www.usenix.org\/conference\/usenixsecurity17\/technical-sessions\/presentation\/wang-tianhao"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1965.10480775"},{"key":"e_1_2_1_30_1","volume-title":"Federated Learning with Differential Privacy: Algorithms and Performance Analysis. CoRR abs\/1911.00222","author":"Wei Kang","year":"2019","unstructured":"Kang Wei , Jun Li , Ming Ding , Chuan Ma , Howard H. Yang , Farhad Farokhi , Shi Jin , Tony Q. S. Quek , and H. Vincent Poor . 2019. Federated Learning with Differential Privacy: Algorithms and Performance Analysis. CoRR abs\/1911.00222 ( 2019 ), 15. arXiv:1911.00222 http:\/\/arxiv.org\/abs\/1911.00222 Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H. Yang, Farhad Farokhi, Shi Jin, Tony Q. S. Quek, and H. Vincent Poor. 2019. Federated Learning with Differential Privacy: Algorithms and Performance Analysis. CoRR abs\/1911.00222 (2019), 15. arXiv:1911.00222 http:\/\/arxiv.org\/abs\/1911.00222"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.5555\/3430915.3442436"},{"key":"e_1_2_1_32_1","volume-title":"Local Differential Privacy and Its Applications: A Comprehensive Survey. CoRR abs\/2008.03686","author":"Yang Mengmeng","year":"2020","unstructured":"Mengmeng Yang , Lingjuan Lyu , Jun Zhao , Tianqing Zhu , and Kwok-Yan Lam . 2020. Local Differential Privacy and Its Applications: A Comprehensive Survey. CoRR abs\/2008.03686 ( 2020 ), 25. arXiv:2008.03686 https:\/\/arxiv.org\/abs\/2008.03686 Mengmeng Yang, Lingjuan Lyu, Jun Zhao, Tianqing Zhu, and Kwok-Yan Lam. 2020. Local Differential Privacy and Its Applications: A Comprehensive Survey. CoRR abs\/2008.03686 (2020), 25. arXiv:2008.03686 https:\/\/arxiv.org\/abs\/2008.03686"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/TDSC.2021.3093711"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3611479.3611523","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,23]],"date-time":"2023-09-23T22:19:26Z","timestamp":1695507566000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3611479.3611523"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7]]},"references-count":33,"journal-issue":{"issue":"11","published-print":{"date-parts":[[2023,7]]}},"alternative-id":["10.14778\/3611479.3611523"],"URL":"https:\/\/doi.org\/10.14778\/3611479.3611523","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2023,7]]},"assertion":[{"value":"2023-08-24","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}