{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T09:15:41Z","timestamp":1781169341255,"version":"3.54.1"},"reference-count":55,"publisher":"Association for Computing Machinery (ACM)","issue":"3","funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62402267"],"award-info":[{"award-number":["62402267"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62341201"],"award-info":[{"award-number":["62341201"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2025,9,3]]},"abstract":"<jats:p>Federated multimodal learning enables decentralized devices with diverse modalities to collaboratively train multimodal models without sharing their raw data. In most existing federated multimodal learning approaches, multimodal data is indispensable. However, in reality, a considerable number of devices can only collect unimodal data, and the data labels are often incomplete. Therefore, this paper proposes FedCMD, a federated multimodal learning approach that enables unimodal devices with missing labels to collaboratively train a multimodal model. To effectively leverage the label-missing samples, FedCMD performs unimodal federated learning first to learn unimodal encoders and make pseudo-labels for the unlabeled samples. Then it calculates and shares the prototypes of various modalities among devices for cross-modal feature alignment. The prototypes are finally served as the complementary of missing modalities to learn a multimodal fusion and classification network. With FedCMD, the learned multimodal network can support both unimodal inputs and multimodal inputs with any missing modalities. Extensive experiments demonstrate the efficacy of FedCMD compared to state-of-the-art baselines.<\/jats:p>","DOI":"10.1145\/3749467","type":"journal-article","created":{"date-parts":[[2025,9,3]],"date-time":"2025-09-03T17:15:45Z","timestamp":1756919745000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Cross-Modal Federated Learning among Unimodal Devices"],"prefix":"10.1145","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3010-3812","authenticated-orcid":false,"given":"Yongheng","family":"Deng","sequence":"first","affiliation":[{"name":"Tsinghua University, Department of Computer Science and Technology, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2093-5693","authenticated-orcid":false,"given":"Ningxin","family":"He","sequence":"additional","affiliation":[{"name":"Nankai University, College of Software, Tianjin, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9702-3417","authenticated-orcid":false,"given":"Xinyi","family":"Li","sequence":"additional","affiliation":[{"name":"Tsinghua University, Department of Computer Science and Technology, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3615-1217","authenticated-orcid":false,"given":"Fan","family":"Wu","sequence":"additional","affiliation":[{"name":"Central South University, School of Computer Science and Engineering, Changsha, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6717-461X","authenticated-orcid":false,"given":"Yaoxue","family":"Zhang","sequence":"additional","affiliation":[{"name":"Tsinghua University, Department of Computer Science and Technology, Beijing, China and Zhongguancun Laboratory, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2782-183X","authenticated-orcid":false,"given":"Ju","family":"Ren","sequence":"additional","affiliation":[{"name":"Tsinghua University, Department of Computer Science and Technology, Beijing, China and Zhongguancun Laboratory, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,9,3]]},"reference":[{"key":"e_1_2_1_1_1","first-page":"3","article-title":"A public domain dataset for human activity recognition using smartphones","volume":"3","author":"Anguita Davide","year":"2013","unstructured":"Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, Jorge Luis Reyes-Ortiz, et al. 2013. A public domain dataset for human activity recognition using smartphones.. In Esann, Vol. 3. 3.","journal-title":"Esann"},{"key":"e_1_2_1_2_1","volume-title":"Multimodal machine learning: A survey and taxonomy","author":"Baltru\u0161aitis Tadas","year":"2018","unstructured":"Tadas Baltru\u0161aitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2018. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence 41, 2 (2018), 423--443."},{"key":"e_1_2_1_3_1","volume-title":"Multimodal Federated Learning with Missing Modality via Prototype Mask and Contrast. arXiv preprint arXiv:2312.13508","author":"Bao Guangyin","year":"2023","unstructured":"Guangyin Bao, Qi Zhang, Duoqian Miao, Zixuan Gong, and Liang Hu. 2023. Multimodal Federated Learning with Missing Modality via Prototype Mask and Contrast. arXiv preprint arXiv:2312.13508 (2023)."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1093\/comjnl\/bxt075"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3534678.3539384"},{"key":"e_1_2_1_6_1","volume-title":"Towards optimal multi-modal federated learning on non-IID data with hierarchical gradient blending","author":"Chen Sijia","unstructured":"Sijia Chen and Baochun Li. 2022. Towards optimal multi-modal federated learning on non-IID data with hierarchical gradient blending. In IEEE INFOCOM. 1469--1478."},{"key":"e_1_2_1_7_1","doi-asserted-by":"crossref","unstructured":"Yixuan Chen Dongsheng Li Peng Zhang Jie Sui Qin Lv Lu Tun and Li Shang. 2022. Cross-modal ambiguity learning for multimodal fake news detection. In ACM WWW. 2897--2905.","DOI":"10.1145\/3485447.3511968"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.23919\/cje.2021.00.093"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v37i6.25891"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3550316"},{"key":"e_1_2_1_11_1","volume-title":"A hierarchical knowledge transfer framework for heterogeneous federated learning","author":"Deng Yongheng","unstructured":"Yongheng Deng, Ju Ren, Cheng Tang, Feng Lyu, Yang Liu, and Yaoxue Zhang. 2023. A hierarchical knowledge transfer framework for heterogeneous federated learning. In IEEE INFOCOM. IEEE, 1--10."},{"key":"e_1_2_1_12_1","doi-asserted-by":"crossref","unstructured":"Yongheng Deng Sheng Yue Tuowei Wang Guanbo Wang Ju Ren and Yaoxue Zhang. 2023. FedINC: An Exemplar-Free Continual Federated Learning Framework with Small Labeled Data. In ACM SenSys. 56--69.","DOI":"10.1145\/3625687.3625800"},{"key":"e_1_2_1_13_1","volume-title":"Fedmultimodal: A benchmark for multimodal federated learning. In ACM SIGKDD. 4035--4045.","author":"Feng Tiantian","year":"2023","unstructured":"Tiantian Feng, Digbalay Bose, Tuo Zhang, Rajat Hebbar, Anil Ramakrishna, Rahul Gupta, Mi Zhang, Salman Avestimehr, and Shrikanth Narayanan. 2023. Fedmultimodal: A benchmark for multimodal federated learning. In ACM SIGKDD. 4035--4045."},{"key":"e_1_2_1_14_1","volume-title":"Distilling the Knowledge in a Neural Network. arXiv preprint arXiv:1503.02531","author":"Hinton Geoffrey","year":"2015","unstructured":"Geoffrey Hinton. 2015. Distilling the Knowledge in a Neural Network. arXiv preprint arXiv:1503.02531 (2015)."},{"key":"e_1_2_1_15_1","volume-title":"Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335","author":"Harry Hsu Tzu-Ming","year":"2019","unstructured":"Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. 2019. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335 (2019)."},{"key":"e_1_2_1_16_1","doi-asserted-by":"crossref","unstructured":"Wenke Huang Mang Ye Zekun Shi He Li and Bo Du. 2023. Rethinking federated learning with domain shift: A prototype view. In 2023 IEEE. In CVPR. 16312--16322.","DOI":"10.1109\/CVPR52729.2023.01565"},{"key":"e_1_2_1_17_1","volume-title":"Federated semi-supervised learning with prototypical networks. arXiv preprint arXiv:2205.13921","author":"Kim Woojung","year":"2022","unstructured":"Woojung Kim, Keondo Park, Kihyuk Sohn, Raphael Shu, and Hyung-Sin Kim. 2022. Federated semi-supervised learning with prototypical networks. arXiv preprint arXiv:2205.13921 (2022)."},{"key":"e_1_2_1_18_1","volume-title":"Yu Qiao, Ye Lin Tun, Minh NH Nguyen, and Choong Seon Hong.","author":"Le Huy Q","year":"2024","unstructured":"Huy Q Le, Chu Myaet Thwal, Yu Qiao, Ye Lin Tun, Minh NH Nguyen, and Choong Seon Hong. 2024. Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modality. arXiv preprint arXiv:2401.13898 (2024)."},{"key":"e_1_2_1_19_1","doi-asserted-by":"crossref","unstructured":"Jingzhi Li Fengling Li Lei Zhu Hui Cui and Jingjing Li. 2023. Prototype-guided knowledge transfer for federated unsupervised cross-modal hashing. In ACM MM. 1013--1022.","DOI":"10.1145\/3581783.3613837"},{"key":"e_1_2_1_20_1","first-page":"9694","article-title":"Align before fuse: Vision and language representation learning with momentum distillation","volume":"34","author":"Li Junnan","year":"2021","unstructured":"Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. NeurIPS 34 (2021), 9694--9705.","journal-title":"NeurIPS"},{"key":"e_1_2_1_21_1","doi-asserted-by":"crossref","unstructured":"Haojie Ma Wenzhong Li Xiao Zhang Songcheng Gao and Sanglu Lu. 2019. AttnSense: Multi-level attention mechanism for multimodal human activity recognition.. In IJCAI. 3109--3115.","DOI":"10.24963\/ijcai.2019\/431"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i3.16330"},{"key":"e_1_2_1_23_1","unstructured":"Brendan McMahan Eider Moore Daniel Ramage Seth Hampson and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics. PMLR 1273--1282."},{"key":"e_1_2_1_24_1","volume-title":"Berkeley mhad: A comprehensive multimodal human action database","author":"Ofli Ferda","unstructured":"Ferda Ofli, Rizwan Chaudhry, Gregorij Kurillo, Ren\u00e9 Vidal, and Ruzena Bajcsy. 2013. Berkeley mhad: A comprehensive multimodal human action database. In IEEE WACV. 53--60."},{"key":"e_1_2_1_25_1","volume-title":"Zhiyuan Xie, Guoliang Xing, and Jianwei Huang.","author":"Ouyang Xiaomin","year":"2022","unstructured":"Xiaomin Ouyang, Xian Shuai, Jiayu Zhou, Ivy Wang Shi, Zhiyuan Xie, Guoliang Xing, and Jianwei Huang. 2022. Cosmo: contrastive fusion learning with small data for multimodal human activity recognition. In MobiCom. 324--337."},{"key":"e_1_2_1_26_1","volume-title":"Harmony: Heterogeneous multi-modal federated learning through disentangled model training. In MobiSys. 530--543.","author":"Ouyang Xiaomin","year":"2023","unstructured":"Xiaomin Ouyang, Zhiyuan Xie, Heming Fu, Sitong Cheng, Li Pan, Neiwen Ling, Guoliang Xing, Jiayu Zhou, and Jianwei Huang. 2023. Harmony: Heterogeneous multi-modal federated learning through disentangled model training. In MobiSys. 530--543."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2021.12.003"},{"key":"e_1_2_1_28_1","doi-asserted-by":"crossref","unstructured":"Attila Reiss and Didier Stricker. 2012. Introducing a new benchmarked dataset for activity monitoring. In ISWC. 108--109.","DOI":"10.1109\/ISWC.2012.13"},{"key":"e_1_2_1_29_1","volume-title":"FLASH: Federated learning for automated selection of high-band mmwave sectors","author":"Salehi Batool","year":"2022","unstructured":"Batool Salehi, Jerry Gu, Debashri Roy, and Kaushik Chowdhury. 2022. FLASH: Federated learning for automated selection of high-band mmwave sectors. In IEEE INFOCOM. IEEE, 1719--1728."},{"key":"e_1_2_1_30_1","doi-asserted-by":"crossref","unstructured":"Philip Schmidt Attila Reiss Robert Duerichen Claus Marberger and Kristof Van Laerhoven. 2018. Introducing wesad a multimodal dataset for wearable stress and affect detection. In ACM ICMI. 400--408.","DOI":"10.1145\/3242969.3242985"},{"key":"e_1_2_1_31_1","doi-asserted-by":"crossref","unstructured":"Qiang Shen Haotian Feng Rui Song Stefano Teso Fausto Giunchiglia Hao Xu et al. 2022. Federated multi-task attention for cross-individual human activity recognition. In IJCAI. 3423--3429.","DOI":"10.24963\/ijcai.2022\/475"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patrec.2021.02.024"},{"key":"e_1_2_1_33_1","volume-title":"Alexey Kurakin, and Chun-Liang Li.","author":"Sohn Kihyuk","year":"2020","unstructured":"Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. 2020. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems 33 (2020), 596--608."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v36i8.20819"},{"key":"e_1_2_1_35_1","volume-title":"Similarity-preserving knowledge distillation","author":"Tung Frederick","unstructured":"Frederick Tung and Greg Mori. 2019. Similarity-preserving knowledge distillation. In IEEE ICCV. 1365--1374."},{"key":"e_1_2_1_36_1","volume-title":"Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks","author":"Wang Lin","year":"2021","unstructured":"Lin Wang and Kuk-Jin Yoon. 2021. Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. IEEE transactions on pattern analysis and machine intelligence 44, 6 (2021), 3048--3068."},{"key":"e_1_2_1_37_1","doi-asserted-by":"crossref","unstructured":"Qi Wang Liang Zhan Paul Thompson and Jiayu Zhou. 2020. Multimodal learning with incomplete modalities by knowledge distillation. In ACM SIGKDD. 1828--1838.","DOI":"10.1145\/3394486.3403234"},{"key":"e_1_2_1_38_1","volume-title":"What makes training multi-modal classification networks hard?","author":"Wang Weiyao","unstructured":"Weiyao Wang, Du Tran, and Matt Feiszli. 2020. What makes training multi-modal classification networks hard?. In IEEE CVPR. 12695--12705."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2022.01.063"},{"key":"e_1_2_1_40_1","doi-asserted-by":"crossref","unstructured":"Baochen Xiong Xiaoshan Yang Yaguang Song Yaowei Wang and Changsheng Xu. 2023. Client-Adaptive Cross-Model Reconstruction Network for Modality-Incomplete Multimodal Federated Learning. In ACM MM. 1241--1249.","DOI":"10.1145\/3581783.3611757"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2023.3275156"},{"key":"e_1_2_1_42_1","volume-title":"Multimodal knowledge expansion","author":"Xue Zihui","unstructured":"Zihui Xue, Sucheng Ren, Zhengqi Gao, and Hang Zhao. 2021. Multimodal knowledge expansion. In IEEE ICCV. 854--863."},{"key":"e_1_2_1_43_1","doi-asserted-by":"crossref","unstructured":"Chengxu Yang Qipeng Wang Mengwei Xu Zhenpeng Chen Kaigui Bian Yunxin Liu and Xuanzhe Liu. 2021. Characterizing impacts of heterogeneity in federated learning upon large-scale smartphone data. In WWW. 935--946.","DOI":"10.1145\/3442381.3449851"},{"key":"e_1_2_1_44_1","volume-title":"Vision-language pre-training with triple contrastive learning","author":"Yang Jinyu","unstructured":"Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. 2022. Vision-language pre-training with triple contrastive learning. In IEEE CVPR. 15671--15680."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3339474"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v36i3.20213"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052577"},{"key":"e_1_2_1_48_1","unstructured":"Qiying Yu Yang Liu Yimu Wang Ke Xu and Jingjing Liu. 2023. Multimodal Federated Learning via Contrastive Representation Ensemble. In ICLR."},{"key":"e_1_2_1_49_1","volume-title":"Multimodal contrastive training for visual representation learning","author":"Yuan Xin","unstructured":"Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faieta. 2021. Multimodal contrastive training for visual representation learning. In IEEE CVPR. 6995--7004."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.23919\/cje.2023.00.288"},{"key":"e_1_2_1_51_1","doi-asserted-by":"crossref","unstructured":"Jinming Zhao Ruichen Li and Qin Jin. 2021. Missing modality imagination network for emotion recognition with uncertain missing modalities. In ACL. 2608--2618.","DOI":"10.18653\/v1\/2021.acl-long.203"},{"key":"e_1_2_1_52_1","volume-title":"Multimodal federated learning on iot data","author":"Zhao Yuchen","unstructured":"Yuchen Zhao, Payam Barnaghi, and Hamed Haddadi. 2022. Multimodal federated learning on iot data. In IEEE\/ACM IoTDI. 43--54."},{"key":"e_1_2_1_53_1","volume-title":"Mirror Gradient: Towards Robust Multimodal Recommender Systems via Exploring Flat Local Minima. In ACM WWW. 3700--3711.","author":"Zhong Shanshan","year":"2024","unstructured":"Shanshan Zhong, Zhongzhan Huang, Daifeng Li, Wushao Wen, Jinghui Qin, and Liang Lin. 2024. Mirror Gradient: Towards Robust Multimodal Recommender Systems via Exploring Flat Local Minima. In ACM WWW. 3700--3711."},{"key":"e_1_2_1_54_1","volume-title":"CrossCLR: Cross-modal contrastive learning for multi-modal video representations","author":"Zolfaghari Mohammadreza","unstructured":"Mohammadreza Zolfaghari, Yi Zhu, Peter Gehler, and Thomas Brox. 2021. CrossCLR: Cross-modal contrastive learning for multi-modal video representations. In IEEE ICCV. 1450--1459."},{"key":"e_1_2_1_55_1","doi-asserted-by":"crossref","unstructured":"Linlin Zong Qiujie Xie Jiahui Zhou Peiran Wu Xianchao Zhang and Bo Xu. 2021. FedCMR: Federated cross-modal retrieval. In ACM SIGIR. 1672--1676.","DOI":"10.1145\/3404835.3462989"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3749467","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,25]],"date-time":"2025-09-25T16:32:50Z","timestamp":1758817970000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3749467"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,3]]},"references-count":55,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,9,3]]}},"alternative-id":["10.1145\/3749467"],"URL":"https:\/\/doi.org\/10.1145\/3749467","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,3]]},"assertion":[{"value":"2025-09-03","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}