{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,18]],"date-time":"2026-03-18T13:51:53Z","timestamp":1773841913476,"version":"3.50.1"},"reference-count":39,"publisher":"MDPI AG","issue":"16","license":[{"start":{"date-parts":[[2021,8,12]],"date-time":"2021-08-12T00:00:00Z","timestamp":1628726400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Emotion recognition is an important research field for human\u2013computer interaction. Audio\u2013video emotion recognition is now attacked with deep neural network modeling tools. In published papers, as a rule, the authors show only cases of the superiority in multi-modality over audio-only or video-only modality. However, there are cases of superiority in uni-modality that can be found. In our research, we hypothesize that for fuzzy categories of emotional events, the within-modal and inter-modal noisy information represented indirectly in the parameters of the modeling neural network impedes better performance in the existing late fusion and end-to-end multi-modal network training strategies. To take advantage of and overcome the deficiencies in both solutions, we define a multi-modal residual perceptron network which performs end-to-end learning from multi-modal network branches, generalizing better multi-modal feature representation. For the proposed multi-modal residual perceptron network and the novel time augmentation for streaming digital movies, the state-of-the-art average recognition rate was improved to 91.4% for the Ryerson Audio\u2013Visual Database of Emotional Speech and Song dataset and to 83.15% for the Crowd-Sourced Emotional Multi Modal Actors dataset. Moreover, the multi-modal residual perceptron network concept shows its potential for multi-modal applications dealing with signal sources not only of optical and acoustical types.<\/jats:p>","DOI":"10.3390\/s21165452","type":"journal-article","created":{"date-parts":[[2021,8,12]],"date-time":"2021-08-12T22:14:36Z","timestamp":1628806476000},"page":"5452","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":23,"title":["Multi-Modal Residual Perceptron Network for Audio\u2013Video Emotion Recognition"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2402-1124","authenticated-orcid":false,"given":"Xin","family":"Chang","sequence":"first","affiliation":[{"name":"Institute of Radioelectronics and Multimedia Technology, Warsaw University of Technology, 00-665 Warsaw, Poland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1917-9181","authenticated-orcid":false,"given":"W\u0142adys\u0142aw","family":"Skarbek","sequence":"additional","affiliation":[{"name":"Institute of Radioelectronics and Multimedia Technology, Warsaw University of Technology, 00-665 Warsaw, Poland"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,8,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"711","DOI":"10.1109\/34.598228","article-title":"Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection","volume":"19","author":"Belhumeur","year":"1997","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-Based Learning Applied to Document Recognition","volume":"86","author":"Lecun","year":"1998","journal-title":"Proc. IEEE"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long Short-Term Memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_4","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention Is All You Need. arXiv."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Neverova, N., Wolf, C., Taylor, G.W., and Nebout, F. (2015). ModDrop: Adaptive multi-modal gesture recognition. arXiv.","DOI":"10.1109\/TPAMI.2015.2461544"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Vielzeuf, V., Pateux, S., and Jurie, F. (2017). Temporal Multimodal Fusion for Video Emotion Classification in the Wild. arXiv.","DOI":"10.1145\/3136755.3143011"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Beard, R., Das, R., Ng, R.W.M., Gopalakrishnan, P.G.K., Eerens, L., Swietojanski, P., and Miksik, O. (2018). Multi-Modal Sequence Fusion via Recursive Attention for Emotion Recognition. Proceedings of the 22nd Conference on Computational Natural Language Learning, Association for Computational Linguistics.","DOI":"10.18653\/v1\/K18-1025"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Ghaleb, E., Popa, M., and Asteriadis, S. (2019, January 3\u20136). Multimodal and Temporal Perception of Audio-visual Cues for Emotion Recognition. Proceedings of the 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII), Cambridge, UK.","DOI":"10.1109\/ACII.2019.8925444"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Zadeh, A., Liang, P.P., Mazumder, N., Poria, S., Cambria, E., and Morency, L.P. (2018). Memory Fusion Network for Multi-view Sequential Learning. arXiv.","DOI":"10.1609\/aaai.v32i1.12021"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Mansouri-Benssassi, E., and Ye, J. (2019, January 14\u201319). Speech Emotion Recognition With Early Visual Cross-modal Enhancement Using Spiking Neural Networks. Proceedings of the 2019 International Joint Conference on Neural Networks (IJCNN), Budapest, Hungary.","DOI":"10.1109\/IJCNN.2019.8852473"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"3030","DOI":"10.1109\/TCSVT.2017.2719043","article-title":"Learning Affective Features with a Hybrid Deep Model for Audio\u2013Visual Emotion Recognition","volume":"28","author":"Zhang","year":"2018","journal-title":"IEEE Trans. Cir. Syst. Video Technol."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Ristea, N., Du\u0163u, L.C., and Radoi, A. (2019, January 10\u201312). Emotion Recognition System from Speech and Visual Information based on Convolutional Neural Networks. Proceedings of the 2019 International Conference on Speech Technology and Human\u2013Computer Dialogue (SpeD), Timisoara, Romania.","DOI":"10.1109\/SPED.2019.8906538"},{"key":"ref_13","unstructured":"Tzinis, E., Wisdom, S., Remez, T., and Hershey, J.R. (2021). Improving On-Screen Sound Separation for Open Domain Videos with Audio\u2013Visual Self-attention. arXiv."},{"key":"ref_14","unstructured":"Wu, Y., Zhu, L., Yan, Y., and Yang, Y. (November, January 27). Dual Attention Matching for Audio\u2013Visual Event Localization. Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV), Seoul, Korea."},{"key":"ref_15","first-page":"37","article-title":"Metric Learning-Based Multimodal Audio\u2013Visual Emotion Recognition","volume":"27","author":"Ghaleb","year":"2020","journal-title":"IEEE MultiMedia"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1109\/TAFFC.2017.2713783","article-title":"Audio\u2013Visual Emotion Recognition in Video Clips","volume":"10","author":"Noroozi","year":"2019","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1016\/j.inffus.2018.09.008","article-title":"Emotion recognition using deep learning approach from audio\u2013visual emotional big data","volume":"49","author":"Hossain","year":"2019","journal-title":"Inf. Fusion"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Ma, F., Zhang, W., Li, Y., Huang, S.L., and Zhang, L. (2020). Learning Better Representations for Audio\u2013Visual Emotion Recognition with Common Information. Appl. Sci., 10.","DOI":"10.3390\/app10207239"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., and Gool, L.V. (2016). Temporal Segment Networks: Towards Good Practices for Deep Action Recognition. arXiv.","DOI":"10.1007\/978-3-319-46484-8_2"},{"key":"ref_20","unstructured":"Simonyan, K., and Zisserman, A. (2014). Two-Stream Convolutional Networks for Action Recognition in Videos. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Carreira, J., and Zisserman, A. (2018). Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. arXiv.","DOI":"10.1109\/CVPR.2017.502"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Wang, W., Tran, D., and Feiszli, M. (2020, January 16\u201318). What Makes Training Multi-Modal Classification Networks Hard?. Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01271"},{"key":"ref_23","unstructured":"Standley, T., Zamir, A.R., Chen, D., Guibas, L., Malik, J., and Savarese, S. (2020). Which Tasks Should Be Learned Together in Multi-task Learning?. arXiv."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1023\/A:1007379606734","article-title":"Multitask Learning","volume":"28","author":"Caruana","year":"1997","journal-title":"Mach. Learn."},{"key":"ref_25","unstructured":"Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., and Finn, C. (2020). Gradient Surgery for Multi-Task Learning. arXiv."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1016\/j.neunet.2014.09.005","article-title":"Challenges in representation learning: A report on three machine learning contests","volume":"64","author":"Goodfellow","year":"2015","journal-title":"Neural Netw."},{"key":"ref_27","unstructured":"Wang, W., Fu, Y., Sun, Q., Chen, T., Cao, C., Zheng, Z., Xu, G., Qiu, H., Jiang, Y.G., and Xue, X. (2020). Learning to Augment Expressions for Few-shot Fine-grained Facial Expression Recognition. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Ng, H.W., Nguyen, V.D., Vonikakis, V., and Winkler, S. (2015). Deep Learning for Emotion Recognition on Small Datasets Using Transfer Learning. Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, Association for Computing Machinery.","DOI":"10.1145\/2818346.2830593"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Dhall, A., Kaur, A., Goecke, R., and Gedeon, T. (2018, January 16\u201320). EmotiW 2018: Audio\u2013Video, Student Engagement and Group-Level Affect Prediction. Proceedings of the 20th ACM International Conference on Multimodal Interaction, Boulder, CO, USA.","DOI":"10.1145\/3242969.3264993"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"335","DOI":"10.1007\/s10579-008-9076-6","article-title":"IEMOCAP: Interactive emotional dyadic motion capture database","volume":"42","author":"Busso","year":"2008","journal-title":"Lang. Resour. Eval."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"and Kwon, S. (2020). A CNN-Assisted Enhanced Audio Signal Processing for Speech Emotion Recognition. Sensors, 20.","DOI":"10.3390\/s20010183"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Latif, S., Rana, R., Qadir, J., and Epps, J. (2017). Variational Autoencoders for Learning Latent Representations of Speech Emotion. arXiv.","DOI":"10.21437\/Interspeech.2018-1568"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"377","DOI":"10.1109\/TAFFC.2014.2336244","article-title":"CREMA-D: Crowd-Sourced Emotional Multimodal Actors Dataset","volume":"5","author":"Cao","year":"2014","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Livingstone, S.R., and Russo, F.A. (2018). The Ryerson Audio\u2013Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English. PLoS ONE, 13.","DOI":"10.1371\/journal.pone.0196391"},{"key":"ref_35","first-page":"1755","article-title":"Dlib-Ml: A Machine Learning Toolkit","volume":"10","author":"King","year":"2009","journal-title":"J. Mach. Learn. Res."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2015). Deep Residual Learning for Image Recognition. arXiv.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_37","unstructured":"Nagrani, A., Chung, J.S., and Zisserman, A. (2021, August 12). VoxCeleb: A Large-Scale Speaker Identification Dataset. Available online: https:\/\/arxiv.org\/pdf\/1706.08612.pdf."},{"key":"ref_38","unstructured":"Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"79861","DOI":"10.1109\/ACCESS.2020.2990405","article-title":"Clustering-Based Speech Emotion Recognition by Incorporating Learned Features and Deep BiLSTM","volume":"8","author":"Mustaqeem","year":"2020","journal-title":"IEEE Access"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/16\/5452\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:45:04Z","timestamp":1760165104000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/16\/5452"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,12]]},"references-count":39,"journal-issue":{"issue":"16","published-online":{"date-parts":[[2021,8]]}},"alternative-id":["s21165452"],"URL":"https:\/\/doi.org\/10.3390\/s21165452","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,8,12]]}}}