{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,25]],"date-time":"2026-01-25T15:45:37Z","timestamp":1769355937602,"version":"3.49.0"},"reference-count":47,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2024,2,15]],"date-time":"2024-02-15T00:00:00Z","timestamp":1707955200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,2,15]],"date-time":"2024-02-15T00:00:00Z","timestamp":1707955200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62272201"],"award-info":[{"award-number":["62272201"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Complex Intell. Syst."],"published-print":{"date-parts":[[2024,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Federated learning has gained significant attention due to its groundbreaking ability to enable distributed learning while maintaining privacy constraints. However, as a consequence of data heterogeneity among decentralized devices, it inherently experiences significant learning degradation and slow convergence speed. Therefore, it is natural to employ the concept of clustering homogeneous clients into the same group, allowing only the model weights within each group to be aggregated. While most existing clustered federated learning methods employ either model gradients or inference outputs as metrics for client partitioning to group similar devices together, heterogeneity may still exist within each cluster. Moreover, there is a scarcity of research exploring the underlying reasons for determining the appropriate timing for clustering, resulting in the common practice of assigning each client to its own individual cluster, particularly in the context of highly non-independent and identically distributed (Non-IID) data. In this paper, we introduce a two-stage decoupling federated learning algorithm with adaptive personalization layers named FedTSDP, where client clustering is performed twice according to inference outputs and model weights, respectively. Hopkins amended sampling is adopted to determine the appropriate timing for clustering and the sampling weight of public unlabeled data. In addition, a simple yet effective approach is developed to adaptively adjust the personalization layers based on varying degrees of data skew. Experimental results show that our proposed method has reliable performance on both IID and non-IID scenarios.<\/jats:p>","DOI":"10.1007\/s40747-024-01342-1","type":"journal-article","created":{"date-parts":[[2024,2,15]],"date-time":"2024-02-15T20:02:21Z","timestamp":1708027341000},"page":"3657-3671","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Federated two-stage decoupling with adaptive personalization layers"],"prefix":"10.1007","volume":"10","author":[{"given":"Hangyu","family":"Zhu","sequence":"first","affiliation":[]},{"given":"Yuxiang","family":"Fan","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9481-9599","authenticated-orcid":false,"given":"Zhenping","family":"Xie","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,2,15]]},"reference":[{"issue":"6","key":"1342_CR1","doi-asserted-by":"publisher","first-page":"3333","DOI":"10.1109\/TCOMM.2023.3253718","volume":"71","author":"MS Al-Abiad","year":"2023","unstructured":"Al-Abiad MS, Obeed M, Hossain MJ, Chaaban A (2023) Decentralized aggregation for energy-efficient federated learning via D2D communications. IEEE Trans Commun 71(6):3333\u20133351. https:\/\/doi.org\/10.1109\/TCOMM.2023.3253718","journal-title":"IEEE Trans Commun"},{"key":"1342_CR2","unstructured":"Arivazhagan MG, Aggarwal V, Singh AK, Choudhary S (2019) Federated learning with personalization layers. arXiv preprint arXiv:1912.00818"},{"key":"1342_CR3","doi-asserted-by":"crossref","unstructured":"Banerjee A, Dave RN (2004) Validating clusters using the Hopkins statistic. In: 2004 IEEE international conference on fuzzy systems (IEEE Cat. No. 04CH37542), vol\u00a01. IEEE, pp 149\u2013153 (2004)","DOI":"10.1109\/FUZZY.2004.1375706"},{"key":"1342_CR4","doi-asserted-by":"crossref","unstructured":"Briggs C, Fan Z, Andras P (2020) Federated learning with hierarchical clustering of local updates to improve training on non-IID data. In: 2020 International joint conference on neural networks (IJCNN). IEEE, pp 1\u20139","DOI":"10.1109\/IJCNN48605.2020.9207469"},{"issue":"1","key":"1342_CR5","doi-asserted-by":"publisher","first-page":"234","DOI":"10.1109\/JSTSP.2022.3231527","volume":"17","author":"YJ Cho","year":"2023","unstructured":"Cho YJ, Wang J, Chirvolu T, Joshi G (2023) Communication-efficient and model-heterogeneous personalized federated learning via clustered knowledge transfer. IEEE J Sel Top Signal Process 17(1):234\u2013247","journal-title":"IEEE J Sel Top Signal Process"},{"key":"1342_CR6","unstructured":"Collins L, Hassani H, Mokhtari A, Shakkottai S (2021) Exploiting shared representations for personalized federated learning. In: International conference on machine learning. PMLR, pp 2089\u20132099"},{"key":"1342_CR7","doi-asserted-by":"publisher","first-page":"146","DOI":"10.1214\/aop\/1176996454","volume":"3","author":"I Csisz\u00e1r","year":"1975","unstructured":"Csisz\u00e1r I (1975) I-divergence geometry of probability distributions and minimization problems. Ann Probab 3:146\u2013158","journal-title":"Ann Probab"},{"issue":"11","key":"1342_CR8","first-page":"2661","volume":"33","author":"M Duan","year":"2021","unstructured":"Duan M, Liu D, Ji X, Wu Y, Liang L, Chen X, Tan Y, Ren A (2021) Flexible clustered federated learning for client-level data distribution shift. IEEE Trans Parallel Distrib Syst 33(11):2661\u20132674","journal-title":"IEEE Trans Parallel Distrib Syst"},{"key":"1342_CR9","first-page":"226","volume":"96","author":"M Ester","year":"1996","unstructured":"Ester M, Kriegel HP, Sander J, Xu X et al (1996) A density-based algorithm for discovering clusters in large spatial databases with noise. KDD 96:226\u2013231","journal-title":"KDD"},{"key":"1342_CR10","doi-asserted-by":"crossref","unstructured":"Fu Y, Liu X, Tang S, Niu J, Huang Z (2021) CIC-FL: enabling class imbalance-aware clustered federated learning over shifted distributions. In: Database systems for advanced applications: 26th international conference, DASFAA 2021, Taipei, April 11\u201314, 2021, Proceedings, Part I 26. Springer, pp 37\u201352","DOI":"10.1007\/978-3-030-73194-6_3"},{"key":"1342_CR11","unstructured":"Ghosh A, Chung J, Yin D, Ramchandran K (2020) An efficient framework for clustered federated learning. Adv Neural Inf Process Syst 33:19586\u201319597"},{"key":"1342_CR12","unstructured":"Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Bach F, Blei D (eds) Proceedings of the 32nd international conference on machine learning, proceedings of machine learning research, vol\u00a037. PMLR, Lille, pp 448\u2013456. https:\/\/proceedings.mlr.press\/v37\/ioffe15.html"},{"key":"1342_CR13","unstructured":"Krizhevsky A, Hinton G (2009) Learning multiple layers of features from tiny images. Master\u2019s thesis, Department of Computer Science, University of Toronto"},{"key":"1342_CR14","doi-asserted-by":"publisher","first-page":"5199","DOI":"10.1007\/s40747-023-01006-6","volume":"9","author":"K Li","year":"2023","unstructured":"Li K, Wang H, Zhang Q (2023) FEDTCR: communication-efficient federated learning via taming computing resources. Complex Intell Syst 9:5199\u20135219","journal-title":"Complex Intell Syst"},{"key":"1342_CR15","doi-asserted-by":"publisher","unstructured":"Li Q, Diao Y, Chen Q, He B (2022) Federated learning on non-IID data silos: an experimental study. In: 2022 IEEE 38th international conference on data engineering (ICDE), pp 965\u2013978. https:\/\/doi.org\/10.1109\/ICDE53745.2022.00077","DOI":"10.1109\/ICDE53745.2022.00077"},{"key":"1342_CR16","unstructured":"Li T, Hu S, Beirami A, Smith V (2021) Ditto: fair and robust federated learning through personalization. In: International conference on machine learning. PMLR, pp 6357\u20136368"},{"key":"1342_CR17","first-page":"429","volume":"2","author":"T Li","year":"2020","unstructured":"Li T, Sahu AK, Zaheer M, Sanjabi M, Talwalkar A, Smith V (2020) Federated optimization in heterogeneous networks. Proc Mach Learn Syst 2:429\u2013450","journal-title":"Proc Mach Learn Syst"},{"key":"1342_CR18","unstructured":"Li X, Jiang M, Zhang X, Kamp M, Dou Q (2021) FedBN: federated learning on non-IID features via local batch normalization. In: International conference on learning representations. https:\/\/openreview.net\/forum?id=6YEQUn0QICG"},{"key":"1342_CR19","doi-asserted-by":"publisher","unstructured":"Li X, Liu N, Chen C, Zheng Z, Li H, Yan Q (2020) Communication-efficient collaborative learning of geo-distributed jointcloud from heterogeneous datasets. In: 2020 IEEE international conference on joint cloud computing, pp 22\u201329. https:\/\/doi.org\/10.1109\/JCC49151.2020.00013","DOI":"10.1109\/JCC49151.2020.00013"},{"issue":"7","key":"1342_CR20","doi-asserted-by":"publisher","first-page":"8423","DOI":"10.1109\/TITS.2021.3081560","volume":"23","author":"Y Li","year":"2022","unstructured":"Li Y, Tao X, Zhang X, Liu J, Xu J (2022) Privacy-preserved federated learning for autonomous driving. IEEE Trans Intell Transp Syst 23(7):8423\u20138434. https:\/\/doi.org\/10.1109\/TITS.2021.3081560","journal-title":"IEEE Trans Intell Transp Syst"},{"key":"1342_CR21","unstructured":"Liang PP, Liu T, Ziyin L, Allen NB, Auerbach RP, Brent D, Salakhutdinov R, Morency LP (2020) Think locally, act globally: federated learning with local and global representations. arXiv preprint arXiv:2001.01523"},{"issue":"1","key":"1342_CR22","doi-asserted-by":"publisher","first-page":"481","DOI":"10.1007\/s11280-022-01046-x","volume":"26","author":"G Long","year":"2023","unstructured":"Long G, Xie M, Shen T, Zhou T, Wang X, Jiang J (2023) Multi-center federated learning: clients clustering for better personalization. World Wide Web 26(1):481\u2013500","journal-title":"World Wide Web"},{"issue":"2","key":"1342_CR23","doi-asserted-by":"publisher","first-page":"2081","DOI":"10.1007\/s40747-022-00895-3","volume":"9","author":"C Lu","year":"2023","unstructured":"Lu C, Ma W, Wang R, Deng S, Wu Y (2023) Federated learning based on stratified sampling and regularization. Complex Intell Syst 9(2):2081\u20132099","journal-title":"Complex Intell Syst"},{"issue":"4","key":"1342_CR24","doi-asserted-by":"publisher","first-page":"1145","DOI":"10.1109\/TPDS.2023.3240767","volume":"34","author":"R Lu","year":"2023","unstructured":"Lu R, Zhang W, Wang Y, Li Q, Zhong X, Yang H, Wang D (2023) Auction-based cluster federated learning in mobile edge computing systems. IEEE Trans Parallel Distrib Syst 34(4):1145\u20131158. https:\/\/doi.org\/10.1109\/TPDS.2023.3240767","journal-title":"IEEE Trans Parallel Distrib Syst"},{"issue":"4","key":"1342_CR25","doi-asserted-by":"crossref","first-page":"1145","DOI":"10.1109\/TPDS.2023.3240767","volume":"34","author":"R Lu","year":"2023","unstructured":"Lu R, Zhang W, Wang Y, Li Q, Zhong X, Yang H, Wang D (2023) Auction-based cluster federated learning in mobile edge computing systems. IEEE Trans Parallel Distrib Syst 34(4):1145\u20131158","journal-title":"IEEE Trans Parallel Distrib Syst"},{"key":"1342_CR26","unstructured":"Luo M, Chen F, Hu D, Zhang Y, Liang J, Feng J (2021) No fear of heterogeneity: classifier calibration for federated learning with non-IID data. In: Ranzato M, Beygelzimer A, Dauphin Y, Liang P, Vaughan JW (eds) Advances in neural information processing systems, vol\u00a034. Curran Associates, Inc., pp 5972\u20135984"},{"key":"1342_CR27","doi-asserted-by":"publisher","first-page":"244","DOI":"10.1016\/j.future.2022.05.003","volume":"135","author":"X Ma","year":"2022","unstructured":"Ma X, Zhu J, Lin Z, Chen S, Qin Y (2022) A state-of-the-art survey on solving non-IID data in federated learning. Future Gener Comput Syst 135:244\u2013258","journal-title":"Future Gener Comput Syst"},{"key":"1342_CR28","unstructured":"McMahan B, Moore E, Ramage D, Hampson S, Arcas BAY (2017) Communication-efficient learning of deep networks from decentralized data. In: Singh A, Zhu J (eds) Proceedings of the 20th international conference on artificial intelligence and statistics, proceedings of machine learning research, vol\u00a054. PMLR, pp 1273\u20131282. https:\/\/proceedings.mlr.press\/v54\/mcmahan17a.html"},{"issue":"2","key":"1342_CR29","doi-asserted-by":"publisher","first-page":"307","DOI":"10.1016\/S0016-0032(96)00063-4","volume":"334","author":"M Men\u00e9ndez","year":"1997","unstructured":"Men\u00e9ndez M, Pardo J, Pardo L, Pardo M (1997) The Jensen\u2013Shannon divergence. J Franklin Inst 334(2):307\u2013318","journal-title":"J Franklin Inst"},{"key":"1342_CR30","doi-asserted-by":"publisher","first-page":"109","DOI":"10.1109\/OJCS.2023.3262203","volume":"4","author":"M Morafah","year":"2023","unstructured":"Morafah M, Vahidian S, Wang W, Lin B (2023) FLIS: clustered federated learning via inference similarity for non-IID data distribution. IEEE Open J Comput Soc 4:109\u2013120","journal-title":"IEEE Open J Comput Soc"},{"issue":"4","key":"1342_CR31","doi-asserted-by":"publisher","first-page":"464","DOI":"10.3390\/e23040464","volume":"23","author":"F Nielsen","year":"2021","unstructured":"Nielsen F (2021) On a variational definition for the Jensen\u2013Shannon symmetrization of distances based on the information radius. Entropy 23(4):464","journal-title":"Entropy"},{"key":"1342_CR32","unstructured":"Pillutla K, Malik K, Mohamed AR, Rabbat M, Sanjabi M, Xiao L (2022) Federated learning with partial model personalization. In: Chaudhuri K, Jegelka S, Song L, Szepesvari C, Niu G, Sabato S (eds) Proceedings of the 39th international conference on machine learning, proceedings of machine learning research, vol 162. PMLR, pp 17716\u201317758. https:\/\/proceedings.mlr.press\/v162\/pillutla22a.html"},{"issue":"4","key":"1342_CR33","doi-asserted-by":"publisher","first-page":"3121","DOI":"10.1007\/s40747-021-00474-y","volume":"8","author":"KV Priya","year":"2022","unstructured":"Priya KV, Peter JD (2022) A federated approach for detecting the chest diseases using densenet for multi-label classification. Complex Intell Syst 8(4):3121\u20133129. https:\/\/doi.org\/10.1007\/s40747-021-00474-y","journal-title":"Complex Intell Syst"},{"key":"1342_CR34","doi-asserted-by":"crossref","unstructured":"Ruan Y, Joe-Wong C (2022) Fedsoft: soft clustered federated learning with proximal local updating. In: Proceedings of the AAAI conference on artificial intelligence, vol\u00a036, pp 8124\u20138131","DOI":"10.1609\/aaai.v36i7.20785"},{"issue":"8","key":"1342_CR35","doi-asserted-by":"publisher","first-page":"3710","DOI":"10.1109\/TNNLS.2020.3015958","volume":"32","author":"F Sattler","year":"2020","unstructured":"Sattler F, M\u00fcller KR, Samek W (2020) Clustered federated learning: model-agnostic distributed multitask optimization under privacy constraints. IEEE Trans Neural Netw Learn Syst 32(8):3710\u20133722","journal-title":"IEEE Trans Neural Netw Learn Syst"},{"key":"1342_CR36","doi-asserted-by":"publisher","unstructured":"Sattler F, Wiedemann S, M\u00fcller KR, Samek W (2020) Robust and communication-efficient federated learning from non-i.i.d. data. IEEE Trans Neural Netw Learn Syst 31(9):3400\u20133413. https:\/\/doi.org\/10.1109\/TNNLS.2019.2944481","DOI":"10.1109\/TNNLS.2019.2944481"},{"key":"1342_CR37","unstructured":"Smith V, Chiang CK, Sanjabi M, Talwalkar AS (2017) Federated multi-task learning. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R (eds) Advances in neural information processing systems, vol\u00a030. Curran Associates, Inc"},{"key":"1342_CR38","doi-asserted-by":"publisher","DOI":"10.1016\/j.cose.2021.102344","volume":"108","author":"P Tian","year":"2021","unstructured":"Tian P, Chen Z, Yu W, Liao W (2021) Towards asynchronous federated learning based threat detection: a DC-ADAM approach. Comput Secur 108:102344","journal-title":"Comput Secur"},{"key":"1342_CR39","doi-asserted-by":"publisher","unstructured":"Tuor T, Wang S, Ko B, Liu C, Leung KK (2021) Overcoming noisy and irrelevant data in federated learning. In: 2020 25th International conference on pattern recognition (ICPR). IEEE Computer Society, Los Alamitos, pp 5020\u20135027. https:\/\/doi.org\/10.1109\/ICPR48806.2021.9412599","DOI":"10.1109\/ICPR48806.2021.9412599"},{"key":"1342_CR40","doi-asserted-by":"publisher","unstructured":"Wang H, Kaplan Z, Niu D, Li B (2020) Optimizing federated learning on non-IID data with reinforcement learning. In: IEEE INFOCOM 2020\u2014IEEE conference on computer communications, pp 1698\u20131707. https:\/\/doi.org\/10.1109\/INFOCOM41043.2020.9155494","DOI":"10.1109\/INFOCOM41043.2020.9155494"},{"issue":"1","key":"1342_CR41","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s40537-016-0043-6","volume":"3","author":"K Weiss","year":"2016","unstructured":"Weiss K, Khoshgoftaar TM, Wang D (2016) A survey of transfer learning. J Big Data 3(1):1\u201340","journal-title":"J Big Data"},{"key":"1342_CR42","doi-asserted-by":"crossref","unstructured":"Yang Z, Liu Y, Zhang S, Zhou K (2023) Personalized federated learning with model interpolation among client clusters and its application in smart home. World Wide Web, pp 1\u201326","DOI":"10.1007\/s11280-022-01132-0"},{"key":"1342_CR43","doi-asserted-by":"crossref","unstructured":"Ye T, Wei S, Cui J, Chen C, Fu Y, Gao M (2023) Robust clustered federated learning. In: International conference on database systems for advanced applications. Springer, New York, pp 677\u2013692","DOI":"10.1007\/978-3-031-30637-2_45"},{"key":"1342_CR44","unstructured":"Zhao Y, Li M, Lai L, Suda N, Civin D, Chandra V (2018) Federated learning with non-IID data. arXiv preprint arXiv:1806.00582"},{"issue":"2","key":"1342_CR45","doi-asserted-by":"publisher","first-page":"1995","DOI":"10.1007\/s40747-022-00894-4","volume":"9","author":"Z Zhong","year":"2023","unstructured":"Zhong Z, Wang J, Bao W, Zhou J, Zhu X, Zhang X (2023) Semi-HFL: semi-supervised federated learning for heterogeneous devices. Complex Intell Syst 9(2):1995\u20132017","journal-title":"Complex Intell Syst"},{"key":"1342_CR46","doi-asserted-by":"publisher","unstructured":"Zhou Y, Ye Q, Lv J (2022) Communication-efficient federated learning with compensated overlap-FEDAVG. IEEE Trans Parallel Distrib Syst 33(1):192\u2013205. https:\/\/doi.org\/10.1109\/TPDS.2021.3090331","DOI":"10.1109\/TPDS.2021.3090331"},{"key":"1342_CR47","doi-asserted-by":"crossref","unstructured":"Zhu H, Xu J, Liu S, Jin Y (2021) Federated learning on non-IID data: a survey. Neurocomputing 465:371\u2013390","DOI":"10.1016\/j.neucom.2021.07.098"}],"container-title":["Complex &amp; Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-024-01342-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s40747-024-01342-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-024-01342-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,11]],"date-time":"2024-11-11T15:18:50Z","timestamp":1731338330000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s40747-024-01342-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,15]]},"references-count":47,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2024,6]]}},"alternative-id":["1342"],"URL":"https:\/\/doi.org\/10.1007\/s40747-024-01342-1","relation":{},"ISSN":["2199-4536","2198-6053"],"issn-type":[{"value":"2199-4536","type":"print"},{"value":"2198-6053","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,2,15]]},"assertion":[{"value":"20 July 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 December 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 February 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"On behalf of all authors, the corresponding author states that there is no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}