{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,3]],"date-time":"2026-04-03T15:12:26Z","timestamp":1775229146908,"version":"3.50.1"},"reference-count":38,"publisher":"Association for Computing Machinery (ACM)","issue":"3","funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62202055 and 62202016"],"award-info":[{"award-number":["62202055 and 62202016"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100021171","name":"Guangdong Basic and Applied Basic Research Foundation","doi-asserted-by":"crossref","award":["2025A1515012843"],"award-info":[{"award-number":["2025A1515012843"]}],"id":[{"id":"10.13039\/501100021171","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Start-up Fund from Beijing Normal University","award":["312200502510"],"award-info":[{"award-number":["312200502510"]}]},{"name":"Internal Fund from Beijing Normal-Hong Kong Baptist University","award":["UICR0400003-24"],"award-info":[{"award-number":["UICR0400003-24"]}]},{"name":"Project of Young Innovative Talents of Guangdong Education Department","award":["2022KQNCX102"],"award-info":[{"award-number":["2022KQNCX102"]}]},{"name":"Interdisciplinary Intelligence SuperComputer Center of Beijing Normal University"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Internet Technol."],"published-print":{"date-parts":[[2025,8,31]]},"abstract":"<jats:p>\n            Multi-modal Federated Learning (FL) is a type of FL that considers utilizing multiple modalities of data to improve overall performance. While multi-modal data brings richer information, it also introduces more significant communication overhead. Reducing this overhead hinges on two key strategies: increasing the convergence speed of the training or reducing the communication overhead in each communication round. However, few studies have considered these two strategies simultaneously and formed a unified optimization framework. Thus, we propose a joint client and modality selection framework to reduce communication overhead. Modality selection executed on each client assigns weights to modalities based on their contribution to training potential, aiming at accelerating the convergence. Client selection executed on the server assigns weights to clients by considering different metrics, especially total training potential after the modality selection. We validate our proposed method on the five widely used open-source datasets, achieving satisfactory accuracy while reducing the total communication overhead to 2.43%\u201314.24% compared to without selection on different datasets, significantly outperforming existing state-of-the-art (SOTA) methods. Code is available at\n            <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" xlink:href=\"https:\/\/github.com\/1643204431\/OCETPMMFL\">https:\/\/github.com\/1643204431\/OCETPMMFL<\/jats:ext-link>\n            .\n          <\/jats:p>","DOI":"10.1145\/3747590","type":"journal-article","created":{"date-parts":[[2025,7,5]],"date-time":"2025-07-05T06:54:29Z","timestamp":1751698469000},"page":"1-31","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Optimizing Communication Efficiency through Training Potential in Multi-Modal Federated Learning"],"prefix":"10.1145","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-2020-869X","authenticated-orcid":false,"given":"Yinghao","family":"Zhang","sequence":"first","affiliation":[{"name":"Advanced Institute of Natural Sciences, Beijing Normal University","place":["Zhuhai, China"]},{"name":"Guangdong Key Lab of AI and Multi-Modal Data Processing, Beijing Normal-Hong Kong Baptist University","place":["Zhuhai, China"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0994-3297","authenticated-orcid":false,"given":"Jianxiong","family":"Guo","sequence":"additional","affiliation":[{"name":"Advanced Institute of Natural Sciences, Beijing Normal University","place":["Zhuhai, China"]},{"name":"Guangdong Key Lab of AI and Multi-Modal Data Processing, Beijing Normal-Hong Kong Baptist University","place":["Zhuhai, China"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8866-4941","authenticated-orcid":false,"given":"Xingjian","family":"Ding","sequence":"additional","affiliation":[{"name":"Faculty of Information Technology, Beijing University of Technology","place":["Beijing, China"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9375-4818","authenticated-orcid":false,"given":"Zhiqing","family":"Tang","sequence":"additional","affiliation":[{"name":"Advanced Institute of Natural Sciences, Beijing Normal University","place":["Zhuhai, China"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4819-621X","authenticated-orcid":false,"given":"Tian","family":"Wang","sequence":"additional","affiliation":[{"name":"Advanced Institute of Natural Sciences, Beijing Normal University","place":["Zhuhai, China"]},{"name":"College of Computer and Data Science, Fuzhou University","place":["Zhuhai, China"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8747-6340","authenticated-orcid":false,"given":"Weili","family":"Wu","sequence":"additional","affiliation":[{"name":"Department of Computer Science, The University of Texas at Dallas","place":["Richardson, United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1000-3937","authenticated-orcid":false,"given":"Weijia","family":"Jia","sequence":"additional","affiliation":[{"name":"Advanced Institute of Natural Sciences, Beijing Normal University","place":["Zhuhai, China"]},{"name":"Guangdong Key Lab of AI and Multi-Modal Data Processing, Beijing Normal-Hong Kong Baptist University","place":["Zhuhai, China"]}]}],"member":"320","published-online":{"date-parts":[[2025,8,19]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Davide Anguita Alessandro Ghio Luca Oneto Xavier Parra Jorge Luis Reyes-Ortiz and others. 2013. A public domain dataset for human activity recognition using smartphones. In Esann. 3\u20134."},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1038\/sdata.2014.53"},{"key":"e_1_3_2_4_2","first-page":"389","volume-title":"Proceedings of the Uncertainty in Artificial Intelligence","author":"Cheng Runxiang","year":"2022","unstructured":"Runxiang Cheng, Gargi Balasubramaniam, Yifei He, Yao-Hung Hubert Tsai, and Han Zhao. 2022. Greedy modality selection via approximate submodular maximization. In Proceedings of the Uncertainty in Artificial Intelligence. PMLR, 389\u2013399."},{"key":"e_1_3_2_5_2","unstructured":"Yae Jee Cho Jianyu Wang and Gauri Joshi. 2020. Client selection in federated learning: Convergence analysis and power-of-choice selection strategies. arXiv:2010.01243. Retrieved from https:\/\/arxiv.org\/abs\/2010.01243"},{"key":"e_1_3_2_6_2","first-page":"13800","article-title":"ActionSense: A multimodal dataset and recording framework for human activities using wearable sensors in a kitchen environment","volume":"35","author":"DelPreto Joseph","year":"2022","unstructured":"Joseph DelPreto, Chao Liu, Yiyue Luo, Michael Foshey, Yunzhu Li, Antonio Torralba, Wojciech Matusik, and Daniela Rus. 2022. ActionSense: A multimodal dataset and recording framework for human activities using wearable sensors in a kitchen environment. Advances in Neural Information Processing Systems 35 (2022), 13800\u201313813.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_7_2","unstructured":"Yunfeng Fan Wenchao Xu Haozhao Wang Penghui Ruan and Song Guo. 2023. Client-wise modality selection for balanced Multi-modal federated learning. arXiv:2401.00403. Retrieved from https:\/\/arxiv.org\/abs\/2401.00403"},{"key":"e_1_3_2_8_2","unstructured":"Yunfeng Fan Wenchao Xu Haozhao Wang Jiaqi Zhu and Song Guo. 2023. balanced Multi-modal federated learning via Cross-Modal infiltration. arXiv:2401.00894. Retrieved from https:\/\/arxiv.org\/abs\/2401.00894"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00983"},{"issue":"47","key":"e_1_3_2_10_2","first-page":"1","article-title":"Efficient modality selection in multimodal learning","volume":"25","author":"He Yifei","year":"2024","unstructured":"Yifei He, Runxiang Cheng, Gargi Balasubramaniam, Yao-Hung Hubert Tsai, and Han Zhao. 2024. Efficient modality selection in multimodal learning. Journal of Machine Learning Research 25, 47 (2024), 1\u201339.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_11_2","first-page":"1","volume-title":"Proceedings of the 2024 IEEE International Conference on Consumer Electronics","author":"Jeon Jangyeong","year":"2024","unstructured":"Jangyeong Jeon, Jungeun Kim, Jinwoo Park, and Junyeong Kim. 2024. MSV: Contribution of modalities based on the shapley value. In Proceedings of the 2024 IEEE International Conference on Consumer Electronics. IEEE, 1\u20136."},{"key":"e_1_3_2_12_2","first-page":"13289","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Joze Hamid Reza Vaezi","year":"2020","unstructured":"Hamid Reza Vaezi Joze, Amirreza Shaban, Michael L Iuzzolino, and Kazuhito Koishida. 2020. MMTM: Multimodal transfer module for CNN fusion. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 13289\u201313299."},{"key":"e_1_3_2_13_2","first-page":"3040","volume-title":"Proceedings of the 2020 42nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society","author":"Ju Ce","year":"2020","unstructured":"Ce Ju, Dashan Gao, Ravikiran Mane, Ben Tan, Yang Liu, and Cuntai Guan. 2020. Federated transfer learning for EEG signal classification. In Proceedings of the 2020 42nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, 3040\u20133045."},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00922"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1186\/s12984-017-0284-4"},{"key":"e_1_3_2_16_2","first-page":"261","volume-title":"Proceedings of the INTERSPEECH","author":"Lau Chun Wai","year":"2004","unstructured":"Chun Wai Lau, Bin Ma, Helen Mei-Ling Meng, Yiu Sang Moon, and Yeung Yam. 2004. Fuzzy logic decision fusion in a multimodal biometric system. In Proceedings of the INTERSPEECH. 261\u2013264."},{"key":"e_1_3_2_17_2","first-page":"429","article-title":"Federated optimization in heterogeneous networks","volume":"2","author":"Li Tian","year":"2020","unstructured":"Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2020. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems 2 (2020), 429\u2013450.","journal-title":"Proceedings of Machine Learning and Systems"},{"key":"e_1_3_2_18_2","first-page":"56","volume-title":"Proceedings of the 2023 IEEE Intl Conf on Parallel and Distributed Processing with Applications, Big Data and Cloud Computing, Sustainable Computing and Communications, Social Computing and Networking","author":"Ling Zhiwei","year":"2023","unstructured":"Zhiwei Ling, Zhihao Yue, Jun Xia, Ting Wang, Mingsong Chen, and Xiang Lian. 2023. FedEntropy: Efficient federated learning for Non-IID scenarios using maximum entropy judgment-based client selection. In Proceedings of the 2023 IEEE Intl Conf on Parallel and Distributed Processing with Applications, Big Data and Cloud Computing, Sustainable Computing and Communications, Social Computing and Networking. IEEE, 56\u201363."},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3581783.3611966"},{"key":"e_1_3_2_20_2","first-page":"326","volume-title":"Proceedings of the European Conference on Computer Vision","author":"Marinov Zdravko","year":"2022","unstructured":"Zdravko Marinov, Alina Roitberg, David Schneider, and Rainer Stiefelhagen. 2022. Modselect: Automatic modality selection for synthetic-to-real domain generalization. In Proceedings of the European Conference on Computer Vision. Springer, 326\u2013346."},{"key":"e_1_3_2_21_2","first-page":"1273","volume-title":"Proceedings of the Artificial intelligence and statistics","author":"McMahan Brendan","year":"2017","unstructured":"Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the Artificial intelligence and statistics. PMLR, 1273\u20131282."},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICC.2019.8761315"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3581791.3596844"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/JBHI.2021.3117575"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSMC.2014.2331920"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP48485.2024.10448255"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2006.09.003"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/INFOCOM48880.2022.9796865"},{"issue":"1","key":"e_1_3_2_29_2","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1038\/s41583-024-00882-2","article-title":"Structural MRI of brain similarity networks","volume":"26","author":"Sebenius Isaac","year":"2025","unstructured":"Isaac Sebenius, Lena Dorfschmidt, Jakob Seidlitz, Aaron Alexander-Bloch, Sarah E. Morgan, and Edward Bullmore. 2025. Structural MRI of brain similarity networks. Nature Reviews Neuroscience 26, 1 (2025), 42\u201359.","journal-title":"Nature Reviews Neuroscience"},{"key":"e_1_3_2_30_2","first-page":"439","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"Tan Weijun","year":"2021","unstructured":"Weijun Tan and Jingfeng Liu. 2021. A 3d cnn network with bert for automatic covid-19 diagnosis from ct-scan images. In Proceedings of the IEEE\/CVF International Conference on Computer Vision. 439\u2013445."},{"key":"e_1_3_2_31_2","article-title":"A scalable federated learning approach for collaborative smart healthcare systems with intermittent clients using medical imaging","author":"Ullah Farhan","year":"2023","unstructured":"Farhan Ullah, Gautam Srivastava, Heng Xiao, Shamsher Ullah, Jerry Chun-Wei Lin, and Yue Zhao. 2023. A scalable federated learning approach for collaborative smart healthcare systems with intermittent clients using medical imaging. IEEE Journal of Biomedical and Health Informatics 28, 6 (2023), 3293\u20133304.","journal-title":"IEEE Journal of Biomedical and Health Informatics"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/INFOCOM41043.2020.9155494"},{"issue":"2","key":"e_1_3_2_33_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3572776","article-title":"Hcms: Hierarchical and conditional modality selection for efficient video recognition","volume":"20","author":"Weng Zejia","year":"2023","unstructured":"Zejia Weng, Zuxuan Wu, Hengduo Li, Jingjing Chen, and Yu-Gang Jiang. 2023. Hcms: Hierarchical and conditional modality selection for efficient video recognition. ACM Transactions on Multimedia Computing, Communications and Applications 20, 2 (2023), 1\u201318.","journal-title":"ACM Transactions on Multimedia Computing, Communications and Applications"},{"issue":"7","key":"e_1_3_2_34_2","doi-asserted-by":"crossref","first-page":"1955","DOI":"10.1109\/TMI.2022.3233405","article-title":"FedMix: Mixed supervised federated learning for medical image segmentation","volume":"42","author":"Wicaksana Jeffry","year":"2022","unstructured":"Jeffry Wicaksana, Zengqiang Yan, Dong Zhang, Xijie Huang, Huimin Wu, Xin Yang, and Kwang-Ting Cheng. 2022. FedMix: Mixed supervised federated learning for medical image segmentation. IEEE Transactions on Medical Imaging 42, 7 (2022), 1955\u20131968.","journal-title":"IEEE Transactions on Medical Imaging"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2022.01.063"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3720539"},{"key":"e_1_3_2_37_2","unstructured":"Liangqi Yuan Dong-Jun Han Su Wang Devesh Upadhyay and Christopher G. Brinton. 2024. Communication-efficient multimodal federated learning: Joint modality and client selection. arXiv:2401.16685. Retrieved from https:\/\/arxiv.org\/abs\/2401.16685"},{"key":"e_1_3_2_38_2","first-page":"43","volume-title":"Proceedings of the 2022 IEEE\/ACM 7th International Conference on Internet-of-Things Design and Implementation","author":"Zhao Yuchen","year":"2022","unstructured":"Yuchen Zhao, Payam Barnaghi, and Hamed Haddadi. 2022. Multimodal federated learning on iot data. In Proceedings of the 2022 IEEE\/ACM 7th International Conference on Internet-of-Things Design and Implementation. IEEE, 43\u201354."},{"key":"e_1_3_2_39_2","first-page":"5590","volume-title":"Proceedings of the ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing","author":"Zhou Yuhao","year":"2024","unstructured":"Yuhao Zhou, Minjia Shi, Yuxin Tian, Yuanxi Li, Qing Ye, and Jiancheng Lv. 2024. Federated CINN clustering for accurate clustered federated learning. In Proceedings of the ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 5590\u20135594."}],"container-title":["ACM Transactions on Internet Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3747590","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,19]],"date-time":"2025-08-19T13:11:41Z","timestamp":1755609101000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3747590"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,19]]},"references-count":38,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,8,31]]}},"alternative-id":["10.1145\/3747590"],"URL":"https:\/\/doi.org\/10.1145\/3747590","relation":{},"ISSN":["1533-5399","1557-6051"],"issn-type":[{"value":"1533-5399","type":"print"},{"value":"1557-6051","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,19]]},"assertion":[{"value":"2024-12-20","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-07-02","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-08-19","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}