{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T15:55:45Z","timestamp":1780502145405,"version":"3.54.1"},"reference-count":37,"publisher":"MDPI AG","issue":"22","license":[{"start":{"date-parts":[[2023,11,16]],"date-time":"2023-11-16T00:00:00Z","timestamp":1700092800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Jilin Provincial Department of Education in China","award":["JJKH20230766KJ"],"award-info":[{"award-number":["JJKH20230766KJ"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Federated learning (FL) is a distributed machine learning paradigm that enables a large number of clients to collaboratively train models without sharing data. However, when the private dataset between clients is not independent and identically distributed (non-IID), the local training objective is inconsistent with the global training objective, which possibly causes the convergence speed of FL to slow down, or even not converge. In this paper, we design a novel FL framework based on deep reinforcement learning (DRL), named FedRLCS. In FedRLCS, we primarily improved the greedy strategy and action space of the double DQN (DDQN) algorithm, enabling the server to select the optimal subset of clients from a non-IID dataset to participate in training, thereby accelerating model convergence and reaching the target accuracy in fewer communication epochs. In simulation experiments, we partition multiple datasets with different strategies to simulate non-IID on local clients. We adopt four models (LeNet-5, MobileNetV2, ResNet-18, ResNet-34) on the four datasets (CIFAR-10, CIFAR-100, NICO, Tiny ImageNet), respectively, and conduct comparative experiments with five state-of-the-art non-IID FL methods. Experimental results show that FedRLCS reduces the number of communication rounds required by 10\u201370% with the same target accuracy without increasing the computation and storage costs for all clients.<\/jats:p>","DOI":"10.3390\/s23229226","type":"journal-article","created":{"date-parts":[[2023,11,16]],"date-time":"2023-11-16T08:19:43Z","timestamp":1700122783000},"page":"9226","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["An Optimization Method for Non-IID Federated Learning Based on Deep Reinforcement Learning"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-5431-5224","authenticated-orcid":false,"given":"Xutao","family":"Meng","sequence":"first","affiliation":[{"name":"School of Computer Science and Engineering, Changchun University of Technology, Changchun 130012, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2907-9811","authenticated-orcid":false,"given":"Yong","family":"Li","sequence":"additional","affiliation":[{"name":"School of Computer Science and Engineering, Changchun University of Technology, Changchun 130012, China"},{"name":"AI Research Institute, Changchun University of Technology, Changchun 130012, China"},{"name":"School of Computer Science and Technology, Jilin University, Changchun 130012, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0788-1448","authenticated-orcid":false,"given":"Jianchao","family":"Lu","sequence":"additional","affiliation":[{"name":"School of Computing, Macquarie University, Sydney, NSW 2109, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xianglin","family":"Ren","sequence":"additional","affiliation":[{"name":"School of Computer Science and Engineering, Changchun University of Technology, Changchun 130012, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,11,16]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1622","DOI":"10.1109\/COMST.2021.3075439","article-title":"Federated learning for internet of things: A comprehensive survey","volume":"23","author":"Nguyen","year":"2021","journal-title":"IEEE Commun. Surv. Tutorials"},{"key":"ref_2","unstructured":"McMahan, B., Moore, E., Ramage, D., Hampson, S., and Arcas, B.A.Y. (2017, January 20\u201322). Communication-efficient learning of deep networks from decentralized data. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Lauderdale, FL, USA."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"22246","DOI":"10.1109\/JIOT.2021.3081480","article-title":"Privacy-preserving federated deep learning for cooperative hierarchical caching in fog computing","volume":"9","author":"Yu","year":"2021","journal-title":"IEEE Internet Things J."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"2046","DOI":"10.1109\/JIOT.2022.3157299","article-title":"Der forecast using privacy preserving federated learning","volume":"10","author":"Venkataramanan","year":"2022","journal-title":"IEEE Internet Things J."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Zhang, C., Liu, X., Zheng, X., Li, R., and Liu, H. (2020, January 23\u201327). Fenghuolun: A federated learning based edge computing platform for cyber-physical systems. Proceedings of the 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), Austin, TX, USA.","DOI":"10.1109\/PerComWorkshops48775.2020.9156259"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"12311","DOI":"10.1109\/JIOT.2021.3135342","article-title":"Bift: A blockchain-based federated learning system for connected and autonomous vehicles","volume":"9","author":"He","year":"2021","journal-title":"IEEE Internet Things J."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"3786","DOI":"10.1109\/JIOT.2022.3176360","article-title":"An efficient content popularity prediction of privacy preserving based on federated learning and wasserstein gan","volume":"10","author":"Wang","year":"2022","journal-title":"IEEE Internet Things J."},{"key":"ref_8","unstructured":"Hsu, T.-M.H., Qi, H., and Brown, M. (2019). Measuring the effects of non-identical data distribution for federated visual classification. arXiv."},{"key":"ref_9","unstructured":"Li, X., Huang, K., Yang, W., Wang, S., and Zhang, Z. (2019). On the convergence of fedavg on non-iid data. arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"371","DOI":"10.1016\/j.neucom.2021.07.098","article-title":"Federated learning on non-iid data: A survey","volume":"465","author":"Zhu","year":"2021","journal-title":"Neurocomputing"},{"key":"ref_11","unstructured":"Karimireddy, S.P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A.T. (2020, January 12\u201318). Scaffold: Stochastic controlled averaging for federated learning. Proceedings of the International Conference on Machine Learning, Vienna, Austria."},{"key":"ref_12","first-page":"429","article-title":"Federated optimization in heterogeneous networks","volume":"2","author":"Li","year":"2020","journal-title":"Proc. Mach. Learn. Syst."},{"key":"ref_13","first-page":"7611","article-title":"Tackling the objective inconsistency problem in heterogeneous federated optimization","volume":"33","author":"Wang","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Li, X.-C., and Zhan, D.-C. (2021, January 14\u201318). Fedrs: Federated learning with restricted softmax for label distribution non-iid data. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, Singapore.","DOI":"10.1145\/3447548.3467254"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Li, Q., He, B., and Song, D. (2021, January 20\u201325). Model-contrastive federated learning. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.01057"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1016\/j.future.2023.01.019","article-title":"Fedproc: Prototypical contrastive federated learning on non-iid data","volume":"143","author":"Mu","year":"2023","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Li, Q., Diao, Y., Chen, Q., and He, B. (2022, January 9\u201312). Federated learning on non-iid data silos: An experimental study. Proceedings of the 2022 IEEE 38th International Conference on Data Engineering (ICDE), Kuala Lumpur, Malaysia.","DOI":"10.1109\/ICDE53745.2022.00077"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1078","DOI":"10.1109\/TCCN.2021.3084406","article-title":"Fast-convergent federated learning with adaptive weighting","volume":"7","author":"Wu","year":"2021","journal-title":"IEEE Trans. Cogn. Commun. Netw."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Zhang, L., Shen, L., Ding, L., Tao, D., and Duan, L.-Y. (2022, January 18\u201324). Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.00993"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"24462","DOI":"10.1109\/ACCESS.2021.3056919","article-title":"Client selection for federated learning with non-iid data in mobile edge computing","volume":"9","author":"Zhang","year":"2021","journal-title":"IEEE Access"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"3099","DOI":"10.1109\/TNSE.2022.3146399","article-title":"Node selection toward faster convergence for federated learning on non-iid data","volume":"9","author":"Wu","year":"2022","journal-title":"IEEE Trans. Netw. Sci. Eng."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Wang, H., Kaplan, Z., Niu, D., and Li, B. (2020, January 6\u20139). Optimizing federated learning on non-iid data with reinforcement learning. Proceedings of the IEEE INFOCOM 2020-IEEE Conference on Computer Communications, Toronto, ON, Canada.","DOI":"10.1109\/INFOCOM41043.2020.9155494"},{"key":"ref_23","unstructured":"Cho, Y.J., Wang, J., and Joshi, G. (2022, January 28\u201330). Towards understanding biased client selection in federated learning. Proceedings of the International Conference on Artificial Intelligence and Statistics, Valencia, Spain."},{"key":"ref_24","unstructured":"Cho, Y.J., Wang, J., and Joshi, G. (2020). Client selection in federated learning: Convergence analysis and power-of-choice selection strategies. arXiv."},{"key":"ref_25","unstructured":"Goetz, J., Malik, K., Bui, D., Moon, S., Liu, H., and Kumar, A. (2019). Active federated learning. arXiv."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Fu, L., Zhang, H., Gao, G., Wang, H., Zhang, M., and Liu, X. (2022). Client selection in federated learning: Principles, challenges, and opportunities. arXiv.","DOI":"10.1109\/JIOT.2023.3299573"},{"key":"ref_27","unstructured":"Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. (2019). The curious case of neural text degeneration. arXiv."},{"key":"ref_28","unstructured":"Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., and Chandra, V. (2018). Federated learning with non-iid data. arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"25506","DOI":"10.1109\/JIOT.2022.3197317","article-title":"Resource-efficient federated learning with non-iid data: An auction theoretic approach","volume":"9","author":"Seo","year":"2022","journal-title":"IEEE Internet Things J."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Yoshida, N., Nishio, T., Morikura, M., Yamamoto, K., and Yonetani, R. (2020, January 7\u201311). Hybrid-fl for wireless networks: Cooperative learning mechanism using non-iid data. Proceedings of the ICC 2020\u20132020 IEEE International Conference On Communications (ICC), Dublin, Ireland.","DOI":"10.1109\/ICC40277.2020.9149323"},{"key":"ref_31","unstructured":"Long, Y., Xue, Z., Chu, L., Zhang, T., Wu, J., Zang, Y., and Du, J. (November, January 29). Fedcd: A classifier debiased federated learning framework for non-iid data. Proceedings of the 31st ACM International Conference on Multimedia, Ottawa, ON, Canada."},{"key":"ref_32","unstructured":"Chen, W., and Zhou, X. (2021, January 14\u201318). Ucbfed: Using reinforcement learning method to tackle the federated optimization problem. Proceedings of the Distributed Applications and Interoperable Systems: 21st IFIP WG 6.1 International Conference, DAIS 2021, Held as Part of the 16th International Federated Conference on Distributed Computing Techniques, DisCoTec 2021, Valletta, Malta."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Xiao, D., Zhan, C., Li, J., and Wu, W. (2023, January 17\u201319). Bi-level sampling: Improved clients selection in heterogeneous settings for federated learning. Proceedings of the 2023 IEEE International Performance, Computing and Communications Conference (IPCCC), Anaheim, CA, USA.","DOI":"10.1109\/IPCCC59175.2023.10253887"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"107383","DOI":"10.1016\/j.patcog.2020.107383","article-title":"Towards non-iid image classification: A dataset and baselines","volume":"110","author":"He","year":"2021","journal-title":"Pattern Recognit."},{"key":"ref_35","unstructured":"Hasselt, H.V., Guez, A., and Silver, D. (2016, January 12\u201317). Deep reinforcement learning with double q-learning. Proceedings of the AAAI Conference on Artificial Intelligence, Phoenix, Arizona, USA."},{"key":"ref_36","unstructured":"Tang, Z., Hu, Z., Shi, S., Cheung, Y.-M., Jin, Y., Ren, Z., and Chu, X. (2021, January 21\u201322). Data resampling for federated learning with non-iid labels. Proceedings of the International Workshop on Federated and Transfer Learning for Data Sparsity and Confidentiality in Conjunction with IJCAI, Montreal, Canada."},{"key":"ref_37","unstructured":"Acar, D.A.E., Zhao, Y., Navarro, R.M., Mattina, M., Whatmough, P.N., and Saligrama, V. (2021). Federated learning based on dynamic regularization. arXiv."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/22\/9226\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T21:24:11Z","timestamp":1760131451000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/22\/9226"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,16]]},"references-count":37,"journal-issue":{"issue":"22","published-online":{"date-parts":[[2023,11]]}},"alternative-id":["s23229226"],"URL":"https:\/\/doi.org\/10.3390\/s23229226","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,16]]}}}