{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T23:27:35Z","timestamp":1771025255635,"version":"3.50.1"},"reference-count":39,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2024,2,22]],"date-time":"2024-02-22T00:00:00Z","timestamp":1708560000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"National Key Research and Development Program of China","award":["2021YFB2900102"],"award-info":[{"award-number":["2021YFB2900102"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62072436"],"award-info":[{"award-number":["62072436"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Innovation Capability Support Program of Shaanxi","award":["2023-CX-TD-08"],"award-info":[{"award-number":["2023-CX-TD-08"]}]},{"name":"Shaanxi Qinchuangyuan \u201cscientists+engineers\u201d team","award":["2023KXJ-040"],"award-info":[{"award-number":["2023KXJ-040"]}]},{"name":"Innovation Funding of ICT, CAS","award":["E261080"],"award-info":[{"award-number":["E261080"]}]},{"name":"Beijing Natural Science Foundation","award":["4212021"],"award-info":[{"award-number":["4212021"]}]},{"name":"Beijing Science and Technology Project","award":["Z211100004121008"],"award-info":[{"award-number":["Z211100004121008"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2024,4,30]]},"abstract":"<jats:p>Federated learning (FL) is a privacy-preserving machine learning paradigm in which the server periodically aggregates local model parameters from cli ents without assembling their private data. Constrained communication and personalization requirements pose severe challenges to FL. Federated distillation (FD) is proposed to simultaneously address the above two problems, which exchanges knowledge between the server and clients, supporting heterogeneous local models while significantly reducing communication overhead. However, most existing FD methods require a proxy dataset, which is often unavailable in reality. A few recent proxy-data-free FD approaches can eliminate the need for additional public data, but suffer from remarkable discrepancy among local knowledge due to client-side model heterogeneity, leading to ambiguous representation on the server and inevitable accuracy degradation. To tackle this issue, we propose a proxy-data-free FD algorithm based on distributed knowledge congruence (FedDKC). FedDKC leverages well-designed refinement strategies to narrow local knowledge differences into an acceptable upper bound, so as to mitigate the negative effects of knowledge incongruence. Specifically, from perspectives of peak probability and Shannon entropy of local knowledge, we design kernel-based knowledge refinement (KKR) and searching-based knowledge refinement (SKR) respectively, and theoretically guarantee that the refined-local knowledge can satisfy an approximately-similar distribution and be regarded as congruent. Extensive experiments conducted on three common datasets demonstrate that our proposed FedDKC significantly outperforms the state-of-the-art on various heterogeneous settings while evidently improving the convergence speed.<\/jats:p>","DOI":"10.1145\/3639369","type":"journal-article","created":{"date-parts":[[2023,12,29]],"date-time":"2023-12-29T22:04:43Z","timestamp":1703887483000},"page":"1-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":20,"title":["Exploring the Distributed Knowledge Congruence in Proxy-data-free Federated Distillation"],"prefix":"10.1145","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8925-4896","authenticated-orcid":false,"given":"Zhiyuan","family":"Wu","sequence":"first","affiliation":[{"name":"Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China and University of Chinese Academy of Sciences, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0260-2692","authenticated-orcid":false,"given":"Sheng","family":"Sun","sequence":"additional","affiliation":[{"name":"Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3228-7371","authenticated-orcid":false,"given":"Yuwei","family":"Wang","sequence":"additional","affiliation":[{"name":"Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2824-9601","authenticated-orcid":false,"given":"Min","family":"Liu","sequence":"additional","affiliation":[{"name":"Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-3958-369X","authenticated-orcid":false,"given":"Quyang","family":"Pan","sequence":"additional","affiliation":[{"name":"Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5947-1374","authenticated-orcid":false,"given":"Junbo","family":"Zhang","sequence":"additional","affiliation":[{"name":"JD iCity, JD Technology, Beijing, China and JD Intelligent Cities Research, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-5453-4394","authenticated-orcid":false,"given":"Zeju","family":"Li","sequence":"additional","affiliation":[{"name":"Beijing University of Posts and Telecommunications, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2277-830X","authenticated-orcid":false,"given":"Qingxiang","family":"Liu","sequence":"additional","affiliation":[{"name":"Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China and University of Chinese Academy of Sciences, Beijing, China"}]}],"member":"320","published-online":{"date-parts":[[2024,2,22]]},"reference":[{"key":"e_1_3_3_2_2","article-title":"Large scale distributed neural network training through online distillation","author":"Anil Rohan","year":"2018","unstructured":"Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George E. Dahl, and Geoffrey E. Hinton. 2018. Large scale distributed neural network training through online distillation. arXiv preprint arXiv:1804.03235 (2018).","journal-title":"arXiv preprint arXiv:1804.03235"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/PERCOMW.2018.8480119"},{"key":"e_1_3_3_4_2","article-title":"Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer","author":"Chang Hongyan","year":"2019","unstructured":"Hongyan Chang, Virat Shejwalkar, Reza Shokri, and Amir Houmansadr. 2019. Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer. arXiv preprint arXiv:1912.11279 (2019).","journal-title":"arXiv preprint arXiv:1912.11279"},{"key":"e_1_3_3_5_2","article-title":"FedGEMS: Federated learning of larger server models via selective knowledge fusion","author":"Cheng Sijie","year":"2021","unstructured":"Sijie Cheng, Jingwen Wu, Yanghua Xiao, and Yang Liu. 2021. FedGEMS: Federated learning of larger server models via selective knowledge fusion. arXiv preprint arXiv:2110.11027 (2021).","journal-title":"arXiv preprint arXiv:2110.11027"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1137\/1019044"},{"key":"e_1_3_3_7_2","article-title":"CINIC-10 is not ImageNet or CIFAR-10","author":"Darlow Luke N.","year":"2018","unstructured":"Luke N. Darlow, Elliot J. Crowley, Antreas Antoniou, and Amos J. Storkey. 2018. CINIC-10 is not ImageNet or CIFAR-10. arXiv preprint arXiv:1810.03505 (2018).","journal-title":"arXiv preprint arXiv:1810.03505"},{"key":"e_1_3_3_8_2","first-page":"14068","article-title":"Group knowledge transfer: Federated learning of large CNNs at the edge","volume":"33","author":"He Chaoyang","year":"2020","unstructured":"Chaoyang He, Murali Annavaram, and Salman Avestimehr. 2020. Group knowledge transfer: Federated learning of large CNNs at the edge. Advances in Neural Information Processing Systems 33 (2020), 14068\u201314080.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_9_2","article-title":"FedML: A research library and benchmark for federated machine learning","author":"He Chaoyang","year":"2020","unstructured":"Chaoyang He, Songze Li, Jinhyun So, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, Li Shen, Peilin Zhao, Yan Kang, Yang Liu, Ramesh Raskar, Qiang Yang, Murali Annavaram, and Salman Avestimehr. 2020. FedML: A research library and benchmark for federated machine learning. arXiv preprint arXiv:2007.13518 (2020).","journal-title":"arXiv preprint arXiv:2007.13518"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00067"},{"key":"e_1_3_3_12_2","article-title":"Distilling the knowledge in a neural network","author":"Hinton Geoffrey","year":"2015","unstructured":"Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015).","journal-title":"arXiv preprint arXiv:1503.02531"},{"key":"e_1_3_3_13_2","unstructured":"Wenke Huang Mang Ye and Bo Du. 2022. https:\/\/github.com\/wenkehuang\/fccl"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00990"},{"key":"e_1_3_3_15_2","article-title":"Distillation-based semi-supervised federated learning for communication-efficient collaborative training with non-IID private data","author":"Itahara Sohei","year":"2020","unstructured":"Sohei Itahara, Takayuki Nishio, Yusuke Koda, Masahiro Morikura, and Koji Yamamoto. 2020. Distillation-based semi-supervised federated learning for communication-efficient collaborative training with non-IID private data. arXiv preprint arXiv:2008.06180 (2020).","journal-title":"arXiv preprint arXiv:2008.06180"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3365109.3368788"},{"issue":"4","key":"e_1_3_3_17_2","first-page":"1355","article-title":"Credit risk assessment from combined bank records using federated learning","volume":"6","author":"Kawa Deep","year":"2019","unstructured":"Deep Kawa, Sunaina Punyani, Priya Nayak, Arpita Karkera, and Varshapriya Jyotinagar. 2019. Credit risk assessment from combined bank records using federated learning. International Research Journal of Engineering and Technology (IRJET) 6, 4 (2019), 1355\u20131358.","journal-title":"International Research Journal of Engineering and Technology (IRJET)"},{"key":"e_1_3_3_18_2","unstructured":"Alex Krizhevsky and Geoffrey Hinton. 2009. Learning multiple layers of features from tiny images. (2009)."},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/5.726791"},{"key":"e_1_3_3_20_2","article-title":"Preservation of the global knowledge by not-true self knowledge distillation in federated learning","author":"Lee Gihun","year":"2021","unstructured":"Gihun Lee, Yongjin Shin, Minchan Jeong, and Se-Young Yun. 2021. Preservation of the global knowledge by not-true self knowledge distillation in federated learning. arXiv preprint arXiv:2106.03097 (2021).","journal-title":"arXiv preprint arXiv:2106.03097"},{"key":"e_1_3_3_21_2","article-title":"FedMD: Heterogenous federated learning via model distillation","author":"Li Daliang","year":"2019","unstructured":"Daliang Li and Junpu Wang. 2019. FedMD: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581 (2019).","journal-title":"arXiv preprint arXiv:1910.03581"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.01465"},{"key":"e_1_3_3_23_2","first-page":"429","article-title":"Federated optimization in heterogeneous networks","volume":"2","author":"Li Tian","year":"2020","unstructured":"Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2020. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems 2 (2020), 429\u2013450.","journal-title":"Proceedings of Machine Learning and Systems"},{"key":"e_1_3_3_24_2","article-title":"Ensemble distillation for robust model fusion in federated learning","author":"Lin Tao","year":"2020","unstructured":"Tao Lin, Lingjing Kong, Sebastian U. Stich, and Martin Jaggi. 2020. Ensemble distillation for robust model fusion in federated learning. arXiv preprint arXiv:2006.07242 (2020).","journal-title":"arXiv preprint arXiv:2006.07242"},{"key":"e_1_3_3_25_2","first-page":"1273","volume-title":"Artificial Intelligence and Statistics","author":"McMahan Brendan","year":"2017","unstructured":"Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics. PMLR, 1273\u20131282."},{"key":"e_1_3_3_26_2","article-title":"Global knowledge distillation in federated learning","author":"Pan Wanning","year":"2021","unstructured":"Wanning Pan and Lichao Sun. 2021. Global knowledge distillation in federated learning. arXiv preprint arXiv:2107.00051 (2021).","journal-title":"arXiv preprint arXiv:2107.00051"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00511"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-020-00323-1"},{"key":"e_1_3_3_29_2","article-title":"FitNets: Hints for thin deep nets","author":"Romero Adriana","year":"2014","unstructured":"Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2014. FitNets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550 (2014).","journal-title":"arXiv preprint arXiv:1412.6550"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3383313.3411528"},{"key":"e_1_3_3_31_2","article-title":"Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks","author":"Wang Lin","year":"2021","unstructured":"Lin Wang and Kuk-Jin Yoon. 2021. Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"e_1_3_3_32_2","article-title":"Spirit distillation: Precise real-time semantic segmentation of road scenes with insufficient data","author":"Wu Zhiyuan","year":"2021","unstructured":"Zhiyuan Wu, Yu Jiang, Chupeng Cui, Zongmin Yang, Xinhui Xue, and Hong Qi. 2021. Spirit distillation: Precise real-time semantic segmentation of road scenes with insufficient data. arXiv preprint arXiv:2103.13733 (2021).","journal-title":"arXiv preprint arXiv:2103.13733"},{"key":"e_1_3_3_33_2","first-page":"553","volume-title":"Knowledge Science, Engineering and Management","author":"Wu Zhiyuan","year":"2021","unstructured":"Zhiyuan Wu, Yu Jiang, Minghao Zhao, Chupeng Cui, Zongmin Yang, Xinhui Xue, and Hong Qi. 2021. Spirit distillation: A model compression method with multi-domain knowledge transfer. In Knowledge Science, Engineering and Management, Han Qiu, Cheng Zhang, Zongming Fei, Meikang Qiu, and Sun-Yuan Kung (Eds.). Springer International Publishing, Cham, 553\u2013565."},{"key":"e_1_3_3_34_2","article-title":"Survey of knowledge distillation in federated edge learning","author":"Wu Zhiyuan","year":"2023","unstructured":"Zhiyuan Wu, Sheng Sun, Yuwei Wang, Min Liu, Xuefeng Jiang, and Runhan Li. 2023. Survey of knowledge distillation in federated edge learning. arXiv preprint arXiv:2301.05849 (2023).","journal-title":"arXiv preprint arXiv:2301.05849"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2023.3289444"},{"key":"e_1_3_3_36_2","article-title":"FedCache: A knowledge cache-driven federated learning architecture for personalized edge intelligence","author":"Wu Zhiyuan","year":"2023","unstructured":"Zhiyuan Wu, Sheng Sun, Yuwei Wang, Min Liu, Wen Wang, Xuefeng Jiang, Bo Gao, and Jinda Lu. 2023. FedCache: A knowledge cache-driven federated learning architecture for personalized edge intelligence. arXiv preprint arXiv:2308.07816 (2023).","journal-title":"arXiv preprint arXiv:2308.07816"},{"key":"e_1_3_3_37_2","article-title":"Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms","author":"Xiao Han","year":"2017","unstructured":"Han Xiao, Kashif Rasul, and Roland Vollgraf. 2017. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017).","journal-title":"arXiv preprint arXiv:1708.07747"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1007\/s41666-020-00082-4"},{"key":"e_1_3_3_39_2","first-page":"arXiv\u20132107","article-title":"Local-global knowledge distillation in heterogeneous federated learning with non-IID data","author":"Yao Dezhong","year":"2021","unstructured":"Dezhong Yao, Wanning Pan, Yutong Dai, Yao Wan, Xiaofeng Ding, Hai Jin, Zheng Xu, and Lichao Sun. 2021. Local-global knowledge distillation in heterogeneous federated learning with non-IID data. arXiv e-prints (2021), arXiv\u20132107.","journal-title":"arXiv e-prints"},{"key":"e_1_3_3_40_2","first-page":"12878","volume-title":"International Conference on Machine Learning","author":"Zhu Zhuangdi","year":"2021","unstructured":"Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. 2021. Data-free knowledge distillation for heterogeneous federated learning. In International Conference on Machine Learning. PMLR, 12878\u201312889."}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3639369","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3639369","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:54:11Z","timestamp":1750287251000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3639369"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,22]]},"references-count":39,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,4,30]]}},"alternative-id":["10.1145\/3639369"],"URL":"https:\/\/doi.org\/10.1145\/3639369","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"value":"2157-6904","type":"print"},{"value":"2157-6912","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,2,22]]},"assertion":[{"value":"2022-10-03","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-12-10","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-02-22","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}