{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,4]],"date-time":"2026-02-04T18:12:31Z","timestamp":1770228751313,"version":"3.49.0"},"reference-count":49,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2023,8,10]],"date-time":"2023-08-10T00:00:00Z","timestamp":1691625600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"NSFC Young Scientists Fund","award":["62006202"],"award-info":[{"award-number":["62006202"]}]},{"DOI":"10.13039\/501100021171","name":"Guangdong Basic and Applied Basic Research Foundation","doi-asserted-by":"crossref","award":["2022A1515011652"],"award-info":[{"award-number":["2022A1515011652"]}],"id":[{"id":"10.13039\/501100021171","id-type":"DOI","asserted-by":"crossref"}]},{"name":"RGC Early Career Scheme","award":["22200720"],"award-info":[{"award-number":["22200720"]}]},{"name":"CAAI-Huawei MindSpore Open Fund, and HKBU CSD Departmental Incentive Grant"},{"name":"NSFC\/Research Grants Council (RGC) Joint Research Scheme","award":["N_HKBU214\/21"],"award-info":[{"award-number":["N_HKBU214\/21"]}]},{"name":"General Research Fund of RGC","award":["12202622 and 12201321"],"award-info":[{"award-number":["12202622 and 12201321"]}]},{"DOI":"10.13039\/501100001747","name":"Hong Kong Baptist University","doi-asserted-by":"crossref","award":["RC-FNRA-IG\/18-19\/SCI\/03"],"award-info":[{"award-number":["RC-FNRA-IG\/18-19\/SCI\/03"]}],"id":[{"id":"10.13039\/501100001747","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100012166","name":"National Key R&D Program of China","doi-asserted-by":"crossref","award":["2022ZD0160702, 2022ZD0160703"],"award-info":[{"award-number":["2022ZD0160702, 2022ZD0160703"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100003399","name":"STCSM","doi-asserted-by":"crossref","award":["22511106101, 22511105700, 21DZ1100100"],"award-info":[{"award-number":["22511106101, 22511105700, 21DZ1100100"]}],"id":[{"id":"10.13039\/501100003399","id-type":"DOI","asserted-by":"crossref"}]},{"name":"111 plan","award":["BP0719010"],"award-info":[{"award-number":["BP0719010"]}]},{"DOI":"10.13039\/501100001809","name":"NSF of China","doi-asserted-by":"crossref","award":["61973162"],"award-info":[{"award-number":["61973162"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"NSF of Jiangsu Province","award":["BZ2021013, BK20220080"],"award-info":[{"award-number":["BZ2021013, BK20220080"]}]},{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"crossref","award":["30920032202, 30921013114"],"award-info":[{"award-number":["30920032202, 30921013114"]}],"id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Knowl. Discov. Data"],"published-print":{"date-parts":[[2024,1,31]]},"abstract":"<jats:p>Federated Learning (FL) learns a global model in a distributional manner, which does not require local clients to share private data. Such merit has drawn lots of attention in the interaction scenarios, where Federated Reinforcement Learning (FRL) emerges as a cross-field research direction focusing on the robust training of agents. Different from FL, the heterogeneity problem in FRL is more challenging because the data depends on the policy of agents and the environment dynamics. FRL learns to interact under the non-stationary environment feedback, while the typical FL methods aim at handling the constant data heterogeneity. In this article, we are among the first attempts to analyze the heterogeneity problem in FRL and propose an off-policy FRL framework. Specifically, a student\u2013teacher\u2013student model learning and fusion method, termed as<jats:italic>Server-Client Collaborative Distillation<\/jats:italic>(SCCD), is introduced. Unlike the traditional FL, we distill all local models on the server side for model fusion. To reduce the variance of the training, a local distillation is also conducted every time the agent receives the global model. Experimentally, we compare SCCD with a range of straightforward combinations between FL methods and RL. The results demonstrate that SCCD has a superior performance in four classical continuous control tasks with non-IID environments.<\/jats:p>","DOI":"10.1145\/3604939","type":"journal-article","created":{"date-parts":[[2023,6,19]],"date-time":"2023-06-19T15:37:19Z","timestamp":1687189039000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":13,"title":["Server-Client Collaborative Distillation for Federated Reinforcement Learning"],"prefix":"10.1145","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1368-6630","authenticated-orcid":false,"given":"Weiming","family":"Mai","sequence":"first","affiliation":[{"name":"Department of CS, Hong Kong Baptist University, Hong Kong SAR, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6115-5194","authenticated-orcid":false,"given":"Jiangchao","family":"Yao","sequence":"additional","affiliation":[{"name":"CMIC, Shanghai Jiao Tong University &amp; Shanghai AI Laboratory, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4092-9856","authenticated-orcid":false,"given":"Gong","family":"Chen","sequence":"additional","affiliation":[{"name":"School of CSE, Nanjing University of Science and Technology, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5390-9053","authenticated-orcid":false,"given":"Ya","family":"Zhang","sequence":"additional","affiliation":[{"name":"CMIC, Shanghai Jiao Tong University &amp; Shanghai AI Laboratory, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7629-4648","authenticated-orcid":false,"given":"Yiu-Ming","family":"Cheung","sequence":"additional","affiliation":[{"name":"Department of CS, Hong Kong Baptist University, Hong Kong SAR, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6338-0958","authenticated-orcid":false,"given":"Bo","family":"Han","sequence":"additional","affiliation":[{"name":"Department of CS, Hong Kong Baptist University, Hong Kong SAR, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,8,10]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Maruan Al-Shedivat Trapit Bansal Yura Burda Ilya Sutskever Igor Mordatch and Pieter Abbeel. 2018. Continuous adaptation via meta-learning in nonstationary and competitive environments. In ICLR OpenReview.net."},{"key":"e_1_3_2_3_2","unstructured":"Aqeel Anwar and Arijit Raychowdhury. 2021. Multi-task federated reinforcement learning with adversaries. CoRR abs\/2103.06473 (2021)."},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1090\/S0002-9904-1954-09848-8"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/INFOCT.2018.8356853"},{"key":"e_1_3_2_6_2","unstructured":"Brendan McMahan Eider Moore Daniel Ramage Seth Hampson and Blaise Ag\u00fcera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In AISTATS (Proceedings of Machine Learning Research) PMLR 1273\u20131282."},{"key":"e_1_3_2_7_2","unstructured":"Greg Brockman Vicki Cheung Ludwig Pettersson Jonas Schneider John Schulman Jie Tang and Wojciech Zaremba. 2016. OpenAI Gym. CoRR abs\/1606.01540 (2016)."},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/MIS.2020.2994942"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA48506.2021.9561593"},{"key":"e_1_3_2_10_2","first-page":"1597","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Chen Ting","year":"2020","unstructured":"Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In Proceedings of the International Conference on Machine Learning. PMLR, 1597\u20131607."},{"key":"e_1_3_2_11_2","unstructured":"Flint Xiaofeng Fan Yining Ma Zhongxiang Dai Wei Jing Cheston Tan and Bryan Kian Hsiang Low. 2021. Fault-tolerant federated reinforcement learning with theoretical guarantee. In NeurIPS 1007\u20131021."},{"key":"e_1_3_2_12_2","first-page":"31","volume-title":"Proceedings of the International Symposium on Information Theory.","author":"Fuglede Bent","year":"2004","unstructured":"Bent Fuglede and Flemming Topsoe. 2004. Jensen-Shannon divergence and Hilbert space embedding. In Proceedings of the International Symposium on Information Theory. IEEE, 31."},{"key":"e_1_3_2_13_2","first-page":"1587","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Fujimoto Scott","year":"2018","unstructured":"Scott Fujimoto, Herke Hoof, and David Meger. 2018. Addressing function approximation error in actor-critic methods. In Proceedings of the International Conference on Machine Learning. PMLR, 1587\u20131596."},{"key":"e_1_3_2_14_2","unstructured":"Scott Fujimoto Herke van Hoof and David Meger. 2018. Addressing function approximation error in actor-critic methods. In ICML (Proceedings of Machine Learning Research) PMLR 1582\u20131591."},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11277-022-09624-y"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1016\/0893-6080(89)90020-8"},{"key":"e_1_3_2_17_2","first-page":"18","volume-title":"Proceedings of the International Conference on Artificial Intelligence and Statistics","author":"Jin Hao","year":"2022","unstructured":"Hao Jin, Yang Peng, Wenhao Yang, Shusen Wang, and Zhihua Zhang. 2022. Federated reinforcement learning with environment heterogeneity. In Proceedings of the International Conference on Artificial Intelligence and Statistics. PMLR, 18\u201337."},{"key":"e_1_3_2_18_2","first-page":"5132","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Karimireddy Sai Praneeth","year":"2020","unstructured":"Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. 2020. Scaffold: Stochastic controlled averaging for federated learning. In Proceedings of the International Conference on Machine Learning. PMLR, 5132\u20135143."},{"key":"e_1_3_2_19_2","first-page":"183","volume-title":"Proceedings of the 54th Annual IEEE\/ACM International Symposium on Microarchitecture","author":"Kim Young Geun","year":"2021","unstructured":"Young Geun Kim and Carole-Jean Wu. 2021. AutoFL: Enabling heterogeneity-aware energy efficient federated learning. In Proceedings of the 54th Annual IEEE\/ACM International Symposium on Microarchitecture. 183\u2013198."},{"key":"e_1_3_2_20_2","unstructured":"Vijay R. Konda and John N. Tsitsiklis. 1999. Actor-Critic Algorithms. In NIPS The MIT Press 1008\u20131014."},{"key":"e_1_3_2_21_2","doi-asserted-by":"crossref","unstructured":"Kwei-Herng Lai Daochen Zha Yuening Li and Xia Hu. 2020. Dual Policy Distillation. In IJCAI ijcai.org 3146\u20133152.","DOI":"10.24963\/ijcai.2020\/435"},{"key":"e_1_3_2_22_2","unstructured":"Daliang Li and Junpu Wang. 2019. FedMD: Heterogenous Federated Learning via Model Distillation. CoRR abs\/1910.03581 (2019)."},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.01057"},{"key":"e_1_3_2_24_2","unstructured":"Tian Li Anit Kumar Sahu Manzil Zaheer Maziar Sanjabi Ameet Talwalkar and Virginia Smith. 2020. Federated Optimization in heterogeneous networks. In MLSys mlsys.org."},{"key":"e_1_3_2_25_2","unstructured":"Xiang Li Kaixuan Huang Wenhao Yang Shusen Wang and Zhihua Zhang. 2020. On the convergence of FedAvg on Non-IID Data. In ICLR OpenReview.net."},{"key":"e_1_3_2_26_2","unstructured":"Xinle Liang Yang Liu Tianjian Chen Ming Liu and Qiang Yang. 2019. Federated transfer reinforcement learning for autonomous driving. CoRR abs\/1910.06001 (2019)."},{"key":"e_1_3_2_27_2","unstructured":"Timothy P. Lillicrap Jonathan J. Hunt Alexander Pritzel Nicolas Heess Tom Erez Yuval Tassa David Silver and Daan Wierstra. 2016. Continuous control with deep reinforcement learning. In ICLR (Poster) ."},{"key":"e_1_3_2_28_2","unstructured":"Tao Lin Lingjing Kong Sebastian U. Stich and Martin Jaggi. 2020. Ensemble distillation for robust model fusion in federated learning. In NeurIPS ."},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/IROS40897.2019.8967908"},{"key":"e_1_3_2_30_2","doi-asserted-by":"crossref","unstructured":"Fan-Ming Luo Shengyi Jiang Yang Yu Zongzhang Zhang and Yi-Feng Zhang. 2022. Adapt to environment sudden changes by learning a context sensitive Policy. In AAAI AAAI Press 7637\u20137646.","DOI":"10.1609\/aaai.v36i7.20730"},{"key":"e_1_3_2_31_2","unstructured":"Mi Luo Fei Chen Dapeng Hu Yifan Zhang Jian Liang and Jiashi Feng. 2021. No fear of heterogeneity: Classifier calibration for federated learning with non-IID data. In NeurIPS 5972\u20135984."},{"key":"e_1_3_2_32_2","unstructured":"Aritra Mitra Rayana H. Jaafar George J. Pappas and Hamed Hassani. 2021. Linear convergence in federated learning: Tackling client Heterogeneity and sparse gradients. In NeurIPS 14606\u201314619."},{"key":"e_1_3_2_33_2","unstructured":"Volodymyr Mnih Koray Kavukcuoglu David Silver Alex Graves Ioannis Antonoglou Daan Wierstra and Martin A. Riedmiller. 2013. Playing atari with deep reinforcement learning. CoRR abs\/1312.5602 (2013)."},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/AIKE.2019.00031"},{"key":"e_1_3_2_35_2","first-page":"4108","article-title":"Deep decentralized multi-task multi-agent reinforcement learning under partial observability","author":"Omidshafiei Shayegan","year":"2017","unstructured":"Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P. How, and John Vian. 2017. Deep decentralized multi-task multi-agent reinforcement learning under partial observability. In Proceedings of the 34th International Conference on Machine Learning. 4108\u20134122. arxiv:1703.06182","journal-title":"Proceedings of the 34th International Conference on Machine Learning"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2009.191"},{"key":"e_1_3_2_37_2","unstructured":"Jiaju Qi Qihao Zhou Lei Lei and Kan Zheng. 2021. Federated reinforcement learning: Techniques applications and open challenges. CoRR abs\/2108.11887 (2021)."},{"key":"e_1_3_2_38_2","unstructured":"Andrei A. Rusu Sergio Gomez Colmenarejo \u00c7aglar G\u00fcl\u00e7ehre Guillaume Desjardins James Kirkpatrick Razvan Pascanu Volodymyr Mnih Koray Kavukcuoglu and Raia Hadsell. 2016. Policy Distillation. In ICLR (Poster) ."},{"key":"e_1_3_2_39_2","first-page":"1889","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Schulman John","year":"2015","unstructured":"John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. 2015. Trust region policy optimization. In Proceedings of the International Conference on Machine Learning. PMLR, 1889\u20131897."},{"key":"e_1_3_2_40_2","unstructured":"John Schulman Filip Wolski Prafulla Dhariwal Alec Radford and Oleg Klimov. 2017. Proximal policy optimization algorithms. CoRR abs\/1707.06347 (2017)."},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1008942012299"},{"key":"e_1_3_2_42_2","article-title":"Policy gradient methods for reinforcement learning with function approximation","author":"Sutton Richard S.","year":"1999","unstructured":"Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. 1999. Policy gradient methods for reinforcement learning with function approximation. In Proceedings of the 12th International Conference on Neural Information Processing Systems.","journal-title":"Proceedings of the 12th International Conference on Neural Information Processing Systems"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1016\/B978-1-55860-307-3.50049-6"},{"key":"e_1_3_2_44_2","doi-asserted-by":"crossref","unstructured":"Jin Wang Jia Hu Jed Mills Geyong Min Ming Xia and Nektarios Georgalas. 2023. Federated ensemble model-based reinforcement learning in edge computing. IEEE Trans. Parallel Distributed Syst. 34 6 (2023) 1848\u20131859.","DOI":"10.1109\/TPDS.2023.3264480"},{"key":"e_1_3_2_45_2","unstructured":"Jianyu Wang Qinghua Liu Hao Liang Gauri Joshi and H. Vincent Poor. 2020. Tackling the objective inconsistency problem in heterogeneous federated optimization. In NeurIPS ."},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3534678.3539181"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3447548.3467097"},{"key":"e_1_3_2_48_2","unstructured":"Jiangchao Yao Shengyu Zhang Yang Yao Feng Wang Jianxin Ma Jianwei Zhang Yunfei Chu Luo Ji Kunyang Jia Tao Shen Anpeng Wu Fengda Zhang Ziqi Tan Kun Kuang Chao Wu Fei Wu Jingren Zhou and Hongxia Yang. 2023. Edge-cloud polarization and collaboration: A comprehensive survey for AI. IEEE Trans. Knowl. Data Eng. 35 7 (2023) 6866\u20136886."},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2023.3236991"},{"key":"e_1_3_2_50_2","unstructured":"Hankz Hankui Zhuo Wenfeng Feng Yufeng Lin Qian Xu and Qiang Yang. 2019. Federated deep reinforcement learning. arXiv e-prints (2019) arXiv-1901."}],"container-title":["ACM Transactions on Knowledge Discovery from Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3604939","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3604939","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:46:37Z","timestamp":1750178797000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3604939"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,10]]},"references-count":49,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,1,31]]}},"alternative-id":["10.1145\/3604939"],"URL":"https:\/\/doi.org\/10.1145\/3604939","relation":{},"ISSN":["1556-4681","1556-472X"],"issn-type":[{"value":"1556-4681","type":"print"},{"value":"1556-472X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,10]]},"assertion":[{"value":"2023-01-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-06-08","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-08-10","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}