{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,11]],"date-time":"2025-12-11T03:05:40Z","timestamp":1765422340548,"version":"3.37.3"},"reference-count":41,"publisher":"Springer Science and Business Media LLC","issue":"8","license":[{"start":{"date-parts":[[2023,8,7]],"date-time":"2023-08-07T00:00:00Z","timestamp":1691366400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,8,7]],"date-time":"2023-08-07T00:00:00Z","timestamp":1691366400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001778","name":"Deakin University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100001778","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Multimed Tools Appl"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Deep learning has been applied to achieve significant progress in emotion recognition from multimedia data. Despite such substantial progress, existing approaches are hindered by insufficient training data, leading to weak generalisation under mismatched conditions. To address these challenges, we propose a learning strategy which jointly transfers emotional knowledge learnt from rich datasets to source-poor datasets. Our method is also able to learn cross-domain features, leading to improved recognition performance. To demonstrate the robustness of the proposed learning strategy, we conducted extensive experiments on several benchmark datasets including eNTERFACE, SAVEE, EMODB, and RAVDESS. Experimental results show that the proposed method surpassed existing transfer learning schemes by a significant margin.<\/jats:p>","DOI":"10.1007\/s11042-023-15441-7","type":"journal-article","created":{"date-parts":[[2023,8,7]],"date-time":"2023-08-07T12:02:11Z","timestamp":1691409731000},"page":"22455-22472","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Deep cross-domain transfer for emotion recognition via joint learning"],"prefix":"10.1007","volume":"83","author":[{"given":"Dung","family":"Nguyen","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2285-2066","authenticated-orcid":false,"given":"Duc Thanh","family":"Nguyen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sridha","family":"Sridharan","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mohamed","family":"Abdelrazek","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Simon","family":"Denman","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Son N.","family":"Tran","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rui","family":"Zeng","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Clinton","family":"Fookes","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2023,8,7]]},"reference":[{"key":"15441_CR1","doi-asserted-by":"crossref","unstructured":"Albanie S, Nagrani A, Vedaldi A, Zisserman A (2018) Emotion recognition in speech using cross-modal transfer in the wild. In: ACM International conference on multimedia (ACMMM), pp 292\u2013301","DOI":"10.1145\/3240508.3240578"},{"key":"15441_CR2","doi-asserted-by":"publisher","first-page":"107111","DOI":"10.1016\/j.patcog.2019.107111","volume":"100","author":"B Bozorgtabar","year":"2020","unstructured":"Bozorgtabar B, Mahapatra D, Thiran J-P (2020) ExprADA: adversarial domain adaptation for facial expression analysis. Pattern Recogn 100:107111","journal-title":"Pattern Recogn"},{"key":"15441_CR3","doi-asserted-by":"crossref","unstructured":"Burkhardt F, Paeschke A, Rolfes M, Sendlmeier W, Weiss B (2005) A database of German emotional speech. In: INTERSPEECH, pp 1517\u20131520","DOI":"10.21437\/Interspeech.2005-446"},{"issue":"4","key":"15441_CR4","doi-asserted-by":"publisher","first-page":"377","DOI":"10.1109\/TAFFC.2014.2336244","volume":"5","author":"H Cao","year":"2014","unstructured":"Cao H, Cooper D G, Keutmann M K, Gur R C, Nenkova A, Verma R (2014) Crema-d: crowd-sourced emotional multimodal actors dataset. IEEE Trans Affect Comput 5(4):377\u2013390","journal-title":"IEEE Trans Affect Comput"},{"key":"15441_CR5","doi-asserted-by":"crossref","unstructured":"Gupta S, Hoffman J, Malik J (2016) Cross modal distillation for supervision transfer. In: IEEE International conference on computer vision and pattern recognition (CVPR), pp 2827\u20132836","DOI":"10.1109\/CVPR.2016.309"},{"key":"15441_CR6","unstructured":"Haq S, Jackson PJB (2009) Speaker-dependent audio-visual emotion recognition. In: International conference on auditory-visual speech processing (AVSP), pp 53\u201358"},{"key":"15441_CR7","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. In: IEEE International conference on computer vision (ICCV) , pp 1026\u20131034","DOI":"10.1109\/ICCV.2015.123"},{"key":"15441_CR8","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: IEEE International conference on computer vision and pattern recognition (CVPR), pp 770\u2013778","DOI":"10.1109\/CVPR.2016.90"},{"key":"15441_CR9","doi-asserted-by":"crossref","unstructured":"He K, Xiangyu Z, Ren S, Sun J (2016) Identity mappings in deep residual networks. In: European conference on computer vision (ECCV), pp 630\u2013645","DOI":"10.1007\/978-3-319-46493-0_38"},{"key":"15441_CR10","unstructured":"Howard A G, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861"},{"key":"15441_CR11","doi-asserted-by":"crossref","unstructured":"Hu G, Liu L, Yuan Y, Yu Z, Hua Y, Zhang Z, Shen F, Shao L, Hospedales T, Robertson N, Yang Y (2018) Deep multi-task learning to recognise subtle facial expressions of mental states. In: European conference on computer vision (ECCV), pp 106\u2013123","DOI":"10.1007\/978-3-030-01258-8_7"},{"key":"15441_CR12","unstructured":"Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In: ICML 2015, vol 37, pp 448\u2013456"},{"key":"15441_CR13","doi-asserted-by":"publisher","first-page":"231","DOI":"10.1016\/j.neucom.2018.12.037","volume":"333","author":"Y Ji","year":"2019","unstructured":"Ji Y, Hu Y, Yang Y, Shen F, Shen H T (2019) Cross-domain facial expression recognition via an intra-category common feature and inter-category distinction feature fusion network. Neurocomputing 333:231\u2013239","journal-title":"Neurocomputing"},{"key":"15441_CR14","doi-asserted-by":"publisher","first-page":"290","DOI":"10.1016\/j.patcog.2018.02.006","volume":"79","author":"H Lei","year":"2018","unstructured":"Lei H, Han T, Zhou F, Yu Z, Qin J, Elazab A, Lei B (2018) A deeply supervised residual network for HEp-2 cell classification via cross-modal transfer learning. Pattern Recogn 79:290\u2013302","journal-title":"Pattern Recogn"},{"issue":"5","key":"15441_CR15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pone.0196391","volume":"13","author":"SR Livingstone","year":"2018","unstructured":"Livingstone S R, Russo F A (2018) The Ryerson audio-visual database of emotional speech and song (RAVDESS): a dynamic, multimodal set of facial and vocal expressions in north american english. PLOS ONE 13(5):1\u201335","journal-title":"PLOS ONE"},{"key":"15441_CR16","unstructured":"Maas A L, Hannun A Y, Ng A Y (2013) Rectifier nonlinearities improve neural network acoustic models. In: International conference on machine learning workshop on deep learning for audio, speech and language processing, pp 1\u20136"},{"key":"15441_CR17","doi-asserted-by":"crossref","unstructured":"Martin O, Kotsia I, Macq B, Pitas I (2006) The eNTERFACE\u2019 05 audio-visual emotion database. In: International conference on data engineering workshops (ICDEW), pp 8\u20138","DOI":"10.1109\/ICDEW.2006.145"},{"key":"15441_CR18","doi-asserted-by":"crossref","unstructured":"Nagrani A, Albanie S, Zisserman A (2018) Seeing voices and hearing faces: cross-modal biometric matching. In: IEEE International conference on computer vision and pattern recognition (CVPR), pp 8427\u20138436","DOI":"10.1109\/CVPR.2018.00879"},{"key":"15441_CR19","doi-asserted-by":"crossref","unstructured":"Nagrani A, Albanie S, Zisserman A (2018) Learnable PINs: cross-modal embeddings for person identity. In: European conference on computer vision (ECCV), pp 73\u201389","DOI":"10.1007\/978-3-030-01261-8_5"},{"key":"15441_CR20","doi-asserted-by":"crossref","unstructured":"Ng H-W, Nguyen V D, Vonikakis V, Winkler S (2015) Deep learning for emotion recognition on small datasets using transfer learning. In: ACM International conference on multimodal interaction (ICMI), pp 443\u2013449","DOI":"10.1145\/2818346.2830593"},{"key":"15441_CR21","doi-asserted-by":"crossref","unstructured":"Nguyen D, Nguyen K, Sridharan S, Abbasnejad I, Dean D, Fookes C (2018) Meta transfer learning for facial emotion recognition. In: International conference on pattern recognition (ICPR), pp 3543\u20133548","DOI":"10.1109\/ICPR.2018.8545411"},{"key":"15441_CR22","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1016\/j.cviu.2018.06.005","volume":"174","author":"D Nguyen","year":"2018","unstructured":"Nguyen D, Nguyen K, Sridharan S, Dean D, Fookes C (2018) Deep spatio-temporal feature fusion with compact bilinear pooling for multimodal emotion recognition. Comput Vis Image Underst 174:33\u201342","journal-title":"Comput Vis Image Underst"},{"key":"15441_CR23","doi-asserted-by":"publisher","first-page":"1313","DOI":"10.1109\/TMM.2021.3063612","volume":"24","author":"D Nguyen","year":"2022","unstructured":"Nguyen D, Nguyen D T, Zeng R, Nguyen T T, Tran S N, Nguyen T, Sridharan S, Fookes C (2022) Deep auto-encoders with sequential learning for multimodal dimensional emotion recognition. IEEE Trans Multimed 24:1313\u20131324","journal-title":"IEEE Trans Multimed"},{"key":"15441_CR24","doi-asserted-by":"publisher","first-page":"10535","DOI":"10.1007\/s00521-023-08248-y","volume":"35","author":"D Nguyen","year":"2023","unstructured":"Nguyen D, Nguyen DT, Sridharan S, Denman S, Nguyen TT, Dean D, Fookes C (2023) Meta-transfer learning for emotion recognition. Neur Comput Applic 35:10535\u201310549","journal-title":"Neur Comput Applic"},{"issue":"1","key":"15441_CR25","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1109\/TAFFC.2017.2713783","volume":"10","author":"F Noroozi","year":"2019","unstructured":"Noroozi F, Marjanovic M, Njegus A, Escalera S, Anbarjafari G (2019) Audio-visual emotion recognition in video clips. IEEE Trans Affect Comput 10(1):60\u201375","journal-title":"IEEE Trans Affect Comput"},{"key":"15441_CR26","doi-asserted-by":"crossref","unstructured":"Ouyang W, Wang X (2013) Joint deep learning for pedestrian detection. In: IEEE International conference on computer vision (ICCV), pp 2056\u20132063","DOI":"10.1109\/ICCV.2013.257"},{"key":"15441_CR27","doi-asserted-by":"crossref","unstructured":"Peng Z, Lu Y, Pan S, Liu Y (2021) Efficient speech emotion recognition using multi-scale CNN and attention. In: IEEE International conference on acoustics, speech and signal processing, pp 3020\u20133024","DOI":"10.1109\/ICASSP39728.2021.9414286"},{"key":"15441_CR28","unstructured":"Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: International conference on learning representations (ICLR), pp 1\u201314"},{"key":"15441_CR29","unstructured":"Sun Y, Chen Y, Wang X, Tang X (2014) Deep learning face representation by joint identification-verification. In: International conference on neural information processing systems (NIPS) , pp 1988\u20131996"},{"key":"15441_CR30","doi-asserted-by":"crossref","unstructured":"Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: IEEE International conference on computer vision and pattern recognition (CVPR), pp 1\u20139","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"15441_CR31","doi-asserted-by":"crossref","unstructured":"Torralba A, Efros A A (2011) Unbiased look at dataset bias. In: IEEE International conference on computer vision and pattern recognition (CVPR), pp 1521\u20131528","DOI":"10.1109\/CVPR.2011.5995347"},{"key":"15441_CR32","doi-asserted-by":"crossref","unstructured":"Wu Y, He K (2018) Group normalization. In: European conference on computer vision (ECCV), pp 3\u201319","DOI":"10.1007\/978-3-030-01261-8_1"},{"key":"15441_CR33","doi-asserted-by":"publisher","first-page":"107335","DOI":"10.1016\/j.patcog.2020.107335","volume":"104","author":"F Wu","year":"2020","unstructured":"Wu F, Jing X-Y, Wu Z, Ji Y, Dong X, Luo X, Huang Q, Wang R (2020) Modality-specific and shared generative adversarial network for cross-modal retrieval. Pattern Recogn 104:107335","journal-title":"Pattern Recogn"},{"key":"15441_CR34","doi-asserted-by":"crossref","unstructured":"Yoon S, Byun S, Dey S, Jung K (2019) Speech emotion recognition using multi-hop attention mechanism. In: IEEE International conference on acoustics, speech and signal processing, pp 2822\u20132826","DOI":"10.1109\/ICASSP.2019.8683483"},{"key":"15441_CR35","doi-asserted-by":"crossref","unstructured":"Zeng J, Shan S, Chen X (2018) Facial expression recognition with inconsistently annotated datasets. In: European Conference on computer vision (ECCV), pp 227\u2013243","DOI":"10.1007\/978-3-030-01261-8_14"},{"issue":"6","key":"15441_CR36","doi-asserted-by":"publisher","first-page":"1576","DOI":"10.1109\/TMM.2017.2766843","volume":"20","author":"S Zhang","year":"2018","unstructured":"Zhang S, Huang T, Gao W (2018) Speech emotion recognition using deep convolutional neural network and discriminant temporal pyramid matching. IEEE Trans Multimed 20(6):1576\u20131590","journal-title":"IEEE Trans Multimed"},{"issue":"10","key":"15441_CR37","doi-asserted-by":"publisher","first-page":"3030","DOI":"10.1109\/TCSVT.2017.2719043","volume":"28","author":"S Zhang","year":"2018","unstructured":"Zhang S, Huang T, Gao W, Tian Q (2018) Learning affective features with a hybrid deep model for audio-visual emotion recognition. IEEE Trans Circuits Syst Video Technol 28(10):3030\u20133043","journal-title":"IEEE Trans Circuits Syst Video Technol"},{"key":"15441_CR38","doi-asserted-by":"crossref","unstructured":"Zhang X, An G, Liu Y (2018) Mask R-CNN with feature pyramid attention for instance segmentation. In: IEEE International conference on signal processing (ICSP), pp 1194\u20131197","DOI":"10.1109\/ICSP.2018.8652371"},{"key":"15441_CR39","doi-asserted-by":"crossref","unstructured":"Zhou Z, Shin J, Zhang L, Gurudu S, Gotway M, Liang J (2017) Fine-tuning convolutional neural networks for biomedical image analysis: actively and incrementally. In: IEEE International conference on computer vision and pattern recognition (CVPR), pp 4761\u20134772","DOI":"10.1109\/CVPR.2017.506"},{"key":"15441_CR40","doi-asserted-by":"crossref","unstructured":"Zhu W, Li X (2022) Speech emotion recognition with global-aware fusion on multi-scale feature representation. In: IEEE International conference on acoustics, speech and signal processing , pp 6437\u20136441","DOI":"10.1109\/ICASSP43922.2022.9747517"},{"key":"15441_CR41","doi-asserted-by":"crossref","unstructured":"Zou H, Si Y, Chen C, Rajan D, Chng E S (2022) Speech emotion recognition with co-attention based multi-level acoustic information. In: IEEE International conference on acoustics, speech and signal processing, pp 7367\u20137371","DOI":"10.1109\/ICASSP43922.2022.9747095"}],"container-title":["Multimedia Tools and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11042-023-15441-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11042-023-15441-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11042-023-15441-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,2,22]],"date-time":"2024-02-22T13:23:00Z","timestamp":1708608180000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11042-023-15441-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,7]]},"references-count":41,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2024,3]]}},"alternative-id":["15441"],"URL":"https:\/\/doi.org\/10.1007\/s11042-023-15441-7","relation":{},"ISSN":["1573-7721"],"issn-type":[{"type":"electronic","value":"1573-7721"}],"subject":[],"published":{"date-parts":[[2023,8,7]]},"assertion":[{"value":"21 December 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 September 2022","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 April 2023","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 August 2023","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no relevant financial or non-financial interests to disclose.The authors have no competing interests to declare that are relevant to the content of this article.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"<!--Emphasis Type='Bold' removed-->Conflict of Interests"}}]}}