{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,5]],"date-time":"2026-06-05T16:13:19Z","timestamp":1780675999583,"version":"3.54.1"},"reference-count":53,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2024,5,17]],"date-time":"2024-05-17T00:00:00Z","timestamp":1715904000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"European Union in the framework of the TEACHING H2020 projects","award":["GA #871385"],"award-info":[{"award-number":["GA #871385"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2024,6,30]]},"abstract":"<jats:p>Emotion recognition has attracted a lot of interest in recent years in various application areas such as healthcare and autonomous driving. Existing approaches to emotion recognition are based on visual, speech, or psychophysiological signals. However, recent studies are looking at multimodal techniques that combine different modalities for emotion recognition. In this work, we address the problem of recognizing the user\u2019s emotion as a driver from unlabeled videos using multimodal techniques. We propose a collaborative training method based on cross-modal distillation, i.e., \u201cFedCMD\u201d (Federated Cross-Modal Distillation). Federated Learning (FL) is an emerging collaborative decentralized learning technique that allows each participant to train their model locally to build a better generalized global model without sharing their data. The main advantage of FL is that only local data is used for training, thus maintaining privacy and providing a secure and efficient emotion recognition system. The local model in FL is trained for each vehicle device with unlabeled video data by using sensor data as a proxy. Specifically, for each local model, we show how driver emotional annotations can be transferred from the sensor domain to the visual domain by using cross-modal distillation. The key idea is based on the observation that a driver\u2019s emotional state indicated by a sensor correlates with facial expressions shown in videos. The proposed \u201cFedCMD\u201d approach is tested on the multimodal dataset \u201cBioVid Emo DB\u201d and achieves state-of-the-art performance. Experimental results show that our approach is robust to non-identically distributed data, achieving 96.67% and 90.83% accuracy in classifying five different emotions with IID (independently and identically distributed) and non-IID data, respectively. Moreover, our model is much more robust to overfitting, resulting in better generalization than the other existing methods.<\/jats:p>","DOI":"10.1145\/3650040","type":"journal-article","created":{"date-parts":[[2024,3,1]],"date-time":"2024-03-01T12:09:30Z","timestamp":1709294970000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":22,"title":["FedCMD: A Federated Cross-modal Knowledge Distillation for Drivers\u2019 Emotion Recognition"],"prefix":"10.1145","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8126-4638","authenticated-orcid":false,"given":"Saira","family":"Bano","sequence":"first","affiliation":[{"name":"University of Pisa, Pisa, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7427-1001","authenticated-orcid":false,"given":"Nicola","family":"Tonellotto","sequence":"additional","affiliation":[{"name":"University of Pisa, Pisa, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3704-4133","authenticated-orcid":false,"given":"Pietro","family":"Cassar\u00e0","sequence":"additional","affiliation":[{"name":"National Research Council, Pisa, Pisa Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8134-7844","authenticated-orcid":false,"given":"Alberto","family":"Gotta","sequence":"additional","affiliation":[{"name":"National Research Council, Pisa, Pisa Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,5,17]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3240508.3240578"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICEngTechnol.2017.8308186"},{"key":"e_1_3_3_4_2","first-page":"125","volume-title":"Advanced Microsystems for Automotive Applications 2016","author":"Ali Mouhannad","year":"2016","unstructured":"Mouhannad Ali, Fadi Al Machot, Ahmad Haj Mosa, and Kyandoghere Kyamakya. 2016. CNN based subject-independent driver emotion recognition system involving physiological signals for ADAS. In Advanced Microsystems for Automotive Applications 2016. Springer, 125\u2013138."},{"key":"e_1_3_3_5_2","doi-asserted-by":"crossref","unstructured":"Yusuf Aytar Carl Vondrick and Antonio Torralba. 2016. SoundNet: Learning sound representations from unlabeled video. Adv. Neural Inf. Process. Syst. 29 (2016).","DOI":"10.1109\/CVPR.2016.18"},{"key":"e_1_3_3_6_2","unstructured":"Jimmy Ba and Rich Caruana. 2014. Do deep nets really need to be deep? Adv. Neural Inf. Process. Syst. 27 (2014)."},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-10-5026-8_1"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/FG.2018.00019"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/SMARTCOMP52413.2021.00086"},{"key":"e_1_3_3_10_2","doi-asserted-by":"crossref","unstructured":"Saira Bano Nicola Tonellotto Pietro Cassar\u00e0 and Alberto Gotta. 2023. Artificial intelligence of things at the edge: Scalable and efficient distributed learning for massive scenarios. Comput. Commun. 205 (2023) 45\u201357.","DOI":"10.1016\/j.comcom.2023.04.010"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/PerComWorkshops53856.2022.9767510"},{"key":"e_1_3_3_12_2","doi-asserted-by":"crossref","unstructured":"Oresti Banos Juan-Manuel Galvez Miguel Damas Hector Pomares and Ignacio Rojas. 2014. Window size impact in human activity recognition. Sensors 14 4 (2014) 6474\u20136499.","DOI":"10.3390\/s140406474"},{"key":"e_1_3_3_13_2","doi-asserted-by":"crossref","unstructured":"Luciano Baresi Danilo Filgueira Mendon\u00e7a Martin Garriga Sam Guinea and Giovanni Quattrocchi. 2019. A unified model for the mobile-edge-cloud continuum. ACM Trans. Internet Technol. 19 2 (2019) 1\u201321.","DOI":"10.1145\/3226644"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/1027933.1027968"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCKE50421.2020.9303716"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/SPAC.2017.8304293"},{"key":"e_1_3_3_17_2","doi-asserted-by":"crossref","unstructured":"Prateek Chhikara Prabhjot Singh Rajkumar Tekchandani Neeraj Kumar and Mohsen Guizani. 2020. Federated learning meets human emotions: A decentralized framework for human\u2013computer interaction for IoT applications. IEEE Internet Things J. 8 8 (2020) 6949\u20136962.","DOI":"10.1109\/JIOT.2020.3037207"},{"key":"e_1_3_3_18_2","first-page":"5961","volume-title":"Proceedings of the IEEE International Conference on Image Processing (ICIP\u201914)","author":"Gao Hua","year":"2014","unstructured":"Hua Gao, Anil Y\u00fcce, and Jean-Philippe Thiran. 2014. Detecting emotional stress from facial expressions for driving safety. In Proceedings of the IEEE International Conference on Image Processing (ICIP\u201914). IEEE, 5961\u20135965."},{"key":"e_1_3_3_19_2","doi-asserted-by":"crossref","unstructured":"Michael Gross. 2016. A planet with two billion cars. Current Biology 26.8 (2016) R307\u2013R310.","DOI":"10.1016\/j.cub.2016.04.019"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.309"},{"key":"e_1_3_3_21_2","doi-asserted-by":"crossref","unstructured":"Jennifer A. Healey and Rosalind W. Picard. 2005. Detecting stress during real-world driving tasks using physiological sensors. IEEE Trans. Intell. Transport. Syst. 6 2 (2005) 156\u2013166.","DOI":"10.1109\/TITS.2005.848368"},{"key":"e_1_3_3_22_2","unstructured":"Geoffrey Hinton Oriol Vinyals and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 2 7 (2015)."},{"key":"e_1_3_3_23_2","first-page":"25","volume-title":"Proceedings of the Asia-Pacific Conference on Communications Technology and Computer Science (ACCTCS\u201921)","author":"Hui Zheng Zack","year":"2021","unstructured":"Zheng Zack Hui, Dingjie Chen, and Zihang Xu. 2021. Federation learning optimization using distillation. In Proceedings of the Asia-Pacific Conference on Communications Technology and Computer Science (ACCTCS\u201921). IEEE, 25\u201328."},{"key":"e_1_3_3_24_2","unstructured":"Arthur Jacot Franck Gabriel and Cl\u00e9ment Hongler. 2018. Neural tangent kernel: Convergence and generalization in neural networks. Adv. Neural Inf. Process. Syst. 31 (2018)."},{"key":"e_1_3_3_25_2","first-page":"5132","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Karimireddy Sai Praneeth","year":"2020","unstructured":"Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. 2020. Scaffold: Stochastic controlled averaging for federated learning. In Proceedings of the International Conference on Machine Learning. PMLR, 5132\u20135143."},{"key":"e_1_3_3_26_2","doi-asserted-by":"crossref","unstructured":"Kyung Hwan Kim Seok Won Bang and Sang Ryong Kim. 2004. Emotion recognition system using short-term monitoring of physiological signals. Med. Biolog. Eng. Comput. 42 3 (2004) 419\u2013427.","DOI":"10.1007\/BF02344719"},{"key":"e_1_3_3_27_2","doi-asserted-by":"crossref","unstructured":"Akshi Kumar Kapil Sharma and Aditi Sharma. 2022. MEmoR: A multimodal emotion recognition using affective biomarkers for smart prediction of emotional health for people analytics in smart industries. Image Vis. Comput. 123 (2022) 104483.","DOI":"10.1016\/j.imavis.2022.104483"},{"key":"e_1_3_3_28_2","doi-asserted-by":"crossref","unstructured":"Wenbo Li Guanzhong Zeng Juncheng Zhang Yan Xu Yang Xing Rui Zhou Gang Guo Yu Shen Dongpu Cao and Fei-Yue Wang. 2021. CogEmoNet: A cognitive-feature-augmented driver emotion recognition model for smart cockpit. IEEE Trans. Computat. Soc. Syst. 9 3 (2021) 667\u2013678.","DOI":"10.1109\/TCSS.2021.3127935"},{"key":"e_1_3_3_29_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Li Xiang","year":"2020","unstructured":"Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. 2020. On the convergence of FedAvg on non-IID data. In Proceedings of the International Conference on Learning Representations. Retrieved from https:\/\/openreview.net\/forum?id=HJxNAnVtDS"},{"key":"e_1_3_3_30_2","doi-asserted-by":"crossref","unstructured":"Jia Zheng Lim James Mountstephens and Jason Teo. 2020. Emotion recognition using eye-tracking: Taxonomy review and current challenges. Sensors 20 8 (2020) 2384.","DOI":"10.3390\/s20082384"},{"key":"e_1_3_3_31_2","doi-asserted-by":"crossref","unstructured":"Yang Liu Keze Wang Guanbin Li and Liang Lin. 2021. Semantics-aware adaptive knowledge distillation for sensor-to-vision action recognition. IEEE Trans. Image Process. 30 (2021) 5573\u20135588.","DOI":"10.1109\/TIP.2021.3086590"},{"key":"e_1_3_3_32_2","first-page":"4820","volume-title":"Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP\u201917)","author":"Lu Liang","year":"2017","unstructured":"Liang Lu, Michelle Guo, and Steve Renals. 2017. Knowledge distillation for small-footprint highway networks. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP\u201917). IEEE, 4820\u20134824."},{"key":"e_1_3_3_33_2","doi-asserted-by":"crossref","unstructured":"Choubeila Maaoui and Alain Pruski. 2010. Emotion recognition through physiological signals for human-machine communication. Cutting Edge Robot. 2010 317-332 (2010) 11.","DOI":"10.5772\/10312"},{"key":"e_1_3_3_34_2","doi-asserted-by":"crossref","unstructured":"Yoshitomo Matsubara Davide Callegaro Sabur Baidya Marco Levorato and Sameer Singh. 2020. Head network distillation: Splitting distilled deep neural networks for resource-constrained edge computing systems. IEEE Access 8 (2020) 212177\u2013212193.","DOI":"10.1109\/ACCESS.2020.3039714"},{"key":"e_1_3_3_35_2","first-page":"1273","volume-title":"Artificial Intelligence and Statistics","author":"McMahan Brendan","year":"2017","unstructured":"Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics. PMLR, 1273\u20131282."},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","unstructured":"Arijit Nandi and Fatos Xhafa. 2022. A federated learning method for real-time emotion state classification from multi-modal streaming. Methods 204 (2022) 340\u2013347. 10.1016\/j.ymeth.2022.03.005","DOI":"10.1016\/j.ymeth.2022.03.005"},{"key":"e_1_3_3_37_2","first-page":"4448","volume-title":"Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP\u201922)","author":"Ni Jianyuan","year":"2022","unstructured":"Jianyuan Ni, Raunak Sarbajna, Yang Liu, Anne HH Ngu, and Yan Yan. 2022. Cross-modal knowledge distillation for vision-to-sensor action recognition. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP\u201922). IEEE, 4448\u20134452."},{"key":"e_1_3_3_38_2","first-page":"5142","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Phuong Mary","year":"2019","unstructured":"Mary Phuong and Christoph Lampert. 2019. Towards understanding knowledge distillation. In Proceedings of the International Conference on Machine Learning. PMLR, 5142\u20135151."},{"key":"e_1_3_3_39_2","doi-asserted-by":"crossref","unstructured":"Srinivasan Ramakrishnan and Ibrahiem M. M. El Emary. 2013. Speech emotion recognition approaches in human computer interaction. Telecommun. Syst. 52 3 (2013) 1467\u20131478.","DOI":"10.1007\/s11235-011-9624-z"},{"key":"e_1_3_3_40_2","unstructured":"H. Tadashi K. Koichi N. Kenta and H. Yuki. 2019. Driver status monitoring system in autonomous driving era. OMRON TECH. 50 (2019)."},{"key":"e_1_3_3_41_2","first-page":"6","volume-title":"Proceedings of the IEEE International Conference on Image Processing (ICIP\u201919)","author":"Thoker Fida Mohammad","year":"2019","unstructured":"Fida Mohammad Thoker and Juergen Gall. 2019. Cross-modal knowledge distillation for action recognition. In Proceedings of the IEEE International Conference on Image Processing (ICIP\u201919). IEEE, 6\u201310."},{"key":"e_1_3_3_42_2","first-page":"55","volume-title":"Proceedings of the 2nd Workshop on Emotion and Computing\u2013Current Research and Future Impact","volume":"1","author":"Tischler Martin A.","year":"2007","unstructured":"Martin A. Tischler, Christian Peter, Matthias Wimmer, and J\u00f6rg Voskamp. 2007. Application of emotion recognition methods in automotive research. In Proceedings of the 2nd Workshop on Emotion and Computing\u2013Current Research and Future Impact, Vol. 1. 55\u201360."},{"key":"e_1_3_3_43_2","doi-asserted-by":"crossref","unstructured":"Nicola Tonellotto Alberto Gotta Franco Maria Nardini Daniele Gadler and Fabrizio Silvestri. 2021. Neural network quantization in federated learning at the edge. Inf. Sci. 575 (2021) 417\u2013436.","DOI":"10.1016\/j.ins.2021.06.039"},{"key":"e_1_3_3_44_2","first-page":"359","volume-title":"Proceedings of the IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops\u201922)","author":"Tsouvalas Vasileios","year":"2022","unstructured":"Vasileios Tsouvalas, Tanir Ozcelebi, and Nirvana Meratnia. 2022. Privacy-preserving speech emotion recognition through semi-supervised federated learning. In Proceedings of the IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops\u201922). IEEE, 359\u2013364."},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICVES.2018.8519595"},{"key":"e_1_3_3_46_2","doi-asserted-by":"crossref","unstructured":"Zhongmin Wang Xiaoxiao Zhou Wenlang Wang and Chen Liang. 2020. Emotion recognition using multimodal deep learning in multiple psychophysiological signals and video. Int. J. Mach. Learn. Cybern. 11 4 (2020) 923\u2013934.","DOI":"10.1007\/s13042-019-01056-8"},{"key":"e_1_3_3_47_2","doi-asserted-by":"crossref","unstructured":"Sheila R. Woody and Bethany A. Teachman. 2000. Intersection of disgust and fear: Normative and pathological views. Clinic. Psychol.: Sci. Pract. 7 3 (2000) 291.","DOI":"10.1093\/clipsy\/7.3.291"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","unstructured":"Huizi Xiao Jun Zhao Qingqi Pei Jie Feng Lei Liu and Weisong Shi. 2022. Vehicle selection and resource optimization for federated learning in vehicular edge computing. IEEE Trans. Intell. Transport. Syst. 23 8 (2022) 11073\u201311087. DOI:10.1109\/TITS.2021.3099597","DOI":"10.1109\/TITS.2021.3099597"},{"key":"e_1_3_3_49_2","unstructured":"Guang Yang Ke Mu Chunhe Song Zhijia Yang and Tierui Gong. 2021. RingFed: Reducing communication costs in federated learning on non-IID data. arXiv preprint arXiv:2107.08873 (2021)."},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/VCIP.2018.8698609"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/SLT.2018.8639583"},{"key":"e_1_3_3_52_2","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1007\/978-3-030-92310-5_24","volume-title":"Proceedings of the International Conference on Neural Information Processing","author":"Zhang Jian-Ming","year":"2021","unstructured":"Jian-Ming Zhang, Xu Yan, Zi-Yi Li, Li-Ming Zhao, Yu-Zhong Liu, Hua-Liang Li, and Bao-Liang Lu. 2021. A cross-subject and cross-modal model for multimodal emotion recognition. In Proceedings of the International Conference on Neural Information Processing. Springer, 203\u2013211."},{"key":"e_1_3_3_53_2","first-page":"1","volume-title":"Proceedings of the IEEE Symposium Series on Computational Intelligence (SSCI\u201916)","author":"Zhang Lin","year":"2016","unstructured":"Lin Zhang, Steffen Walter, Xueyao Ma, Philipp Werner, Ayoub Al-Hamadi, Harald C. Traue, and Sascha Gruss. 2016. \u201cBioVid Emo DB\u201d: A multimodal database for emotion analyses validated by subjective ratings. In Proceedings of the IEEE Symposium Series on Computational Intelligence (SSCI\u201916). IEEE, 1\u20136."},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00768"}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3650040","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3650040","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:03:43Z","timestamp":1750291423000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3650040"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,17]]},"references-count":53,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2024,6,30]]}},"alternative-id":["10.1145\/3650040"],"URL":"https:\/\/doi.org\/10.1145\/3650040","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"value":"2157-6904","type":"print"},{"value":"2157-6912","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,17]]},"assertion":[{"value":"2023-04-11","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-02-19","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-05-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}