{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,17]],"date-time":"2026-01-17T19:37:04Z","timestamp":1768678624001,"version":"3.49.0"},"reference-count":65,"publisher":"Wiley","issue":"1","license":[{"start":{"date-parts":[[2023,10,17]],"date-time":"2023-10-17T00:00:00Z","timestamp":1697500800000},"content-version":"vor","delay-in-days":289,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62176095"],"award-info":[{"award-number":["62176095"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62006049"],"award-info":[{"award-number":["62006049"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004000","name":"Guangzhou Municipal Science and Technology Program key projects","doi-asserted-by":"publisher","award":["2023B03J1336"],"award-info":[{"award-number":["2023B03J1336"]}],"id":[{"id":"10.13039\/501100004000","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004000","name":"Guangzhou Municipal Science and Technology Program key projects","doi-asserted-by":"publisher","award":["2023A03J0316"],"award-info":[{"award-number":["2023A03J0316"]}],"id":[{"id":"10.13039\/501100004000","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100015956","name":"Special Project for Research and Development in Key areas of Guangdong Province","doi-asserted-by":"publisher","award":["2020B1111120001"],"award-info":[{"award-number":["2020B1111120001"]}],"id":[{"id":"10.13039\/501100015956","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100021171","name":"Basic and Applied Basic Research Foundation of Guangdong Province","doi-asserted-by":"publisher","award":["2023A1515010939"],"award-info":[{"award-number":["2023A1515010939"]}],"id":[{"id":"10.13039\/501100021171","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100010226","name":"Department of Education of Guangdong Province","doi-asserted-by":"publisher","award":["2022KTSCX068"],"award-info":[{"award-number":["2022KTSCX068"]}],"id":[{"id":"10.13039\/501100010226","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100010226","name":"Department of Education of Guangdong Province","doi-asserted-by":"publisher","award":["2021ZDZX1079"],"award-info":[{"award-number":["2021ZDZX1079"]}],"id":[{"id":"10.13039\/501100010226","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["International Journal of Intelligent Systems"],"published-print":{"date-parts":[[2023,1]]},"abstract":"<jats:p>Speech emotion recognition has lots of applications such as human\u2010computer interaction and health management. The current methods are challenged with the problems of fuzzy decision boundary and imbalance between difficult and easy samples in the training data. This paper first proposes an additive angle penalty focus loss function (APFL), which strictly refines the fuzzy decision boundary by introducing angle penalty factors to improve the compactness within the class and enlarge the distance between classes. It also assigns the larger loss to difficult samples to make the model pay more attention to them, as they are easily misclassified. Simultaneously, due to the lack of training samples, the framework of multimodal and multitask learning with APFL is further proposed, which extracts spectrogram features by deep neural network, text features by the pretrained language model, and audio features by the pretrained sound model. It uses the gender recognition as an auxiliary task. The experimental results verify the effectiveness of the proposed loss function and framework.<\/jats:p>","DOI":"10.1155\/2023\/3662839","type":"journal-article","created":{"date-parts":[[2023,10,17]],"date-time":"2023-10-17T23:05:54Z","timestamp":1697583954000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Multimodal and Multitask Learning with Additive Angular Penalty Focus Loss for Speech Emotion Recognition"],"prefix":"10.1155","volume":"2023","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9709-1126","authenticated-orcid":false,"given":"Guihua","family":"Wen","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2096-5219","authenticated-orcid":false,"given":"Sheng","family":"Ye","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0463-8178","authenticated-orcid":false,"given":"Huihui","family":"Li","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7258-8323","authenticated-orcid":false,"given":"Pengcheng","family":"Wen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1764-5043","authenticated-orcid":false,"given":"Yuhan","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"311","published-online":{"date-parts":[[2023,10,17]]},"reference":[{"key":"e_1_2_10_1_2","doi-asserted-by":"publisher","DOI":"10.1109\/taslp.2022.3145287"},{"key":"e_1_2_10_2_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2022.109589"},{"key":"e_1_2_10_3_2","doi-asserted-by":"crossref","unstructured":"WenX.-C. YeJ.-X. LuoY. XuY. WangX.-Z. WuC.-L. andLiuK.-H. CTL-MTNet: A novel CapsNet and transfer learning-based mixed task net for the single-corpus and cross-corpus speech emotion recognition Proceedings of the IJCAI International Joint Conference on Artificial Intelligence July 2022 Vienna Austria 2305\u20132311.","DOI":"10.24963\/ijcai.2022\/320"},{"key":"e_1_2_10_4_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.specom.2019.09.002"},{"key":"e_1_2_10_5_2","doi-asserted-by":"crossref","unstructured":"MaoS. TaoD. ZhangG. ChingP. C. andLeeT. Revisiting hidden Markov models for speech emotion recognition Proceedings of the IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP) May 2019 Brighton UK 6715\u20136719.","DOI":"10.1109\/ICASSP.2019.8683172"},{"key":"e_1_2_10_6_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.specom.2022.02.007"},{"key":"e_1_2_10_7_2","doi-asserted-by":"crossref","unstructured":"SattA. RozenbergS. andHooryR. Efficient emotion recognition from speech using deep learning on spectrograms Proceedings of the Annual Conference of the International Speech Communication Association (InterSpeech) August 2017 Stockholm Sweden 1089\u20131093.","DOI":"10.21437\/Interspeech.2017-200"},{"key":"e_1_2_10_8_2","doi-asserted-by":"crossref","unstructured":"LiliG. LongbiaoW. JianwuD. LinjuanZ. HaotianG. andXiangangL. Speech emotion recognition by combining amplitude and phase information using convolutional neural network Proceedings of the Annual Conference of the International Speech Communication Association (InterSpeech) September 2018 Hyderabad India 1611\u20131615.","DOI":"10.21437\/Interspeech.2018-2156"},{"key":"e_1_2_10_9_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.specom.2022.07.005"},{"key":"e_1_2_10_10_2","unstructured":"JiaxinY. Xin-chengW. YujieW. YongX. KunhongL. andHongmingS. Temporal modeling matters: a novel temporal emotional modeling approach for speech emotion recognition 2022 https:\/\/arxiv.org\/abs\/2211.08233."},{"key":"e_1_2_10_11_2","unstructured":"RunnanL. ZhiyongW. JiaJ. ShengZ. andHelenM. Dilated residual network with multi-head self-attention for speech emotion recognition Proceedings of the IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP) May 2019 Brighton UK 6675\u20136679."},{"key":"e_1_2_10_12_2","doi-asserted-by":"crossref","unstructured":"JiaxingL. ZhileiL. LongbiaoW. LiliG. andJianwuD. Speech emotion recognition with local-global aware deep representation learning Proceedings of the IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP) May 2020 Barcelona Spain 7174\u20137178.","DOI":"10.1109\/ICASSP40776.2020.9053192"},{"key":"e_1_2_10_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/lsp.2018.2860246"},{"key":"e_1_2_10_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/tmm.2017.2766843"},{"key":"e_1_2_10_15_2","unstructured":"LiR. ZhaoJ. andJinQ. Speech emotion recognition via multi-level cross-modal distillation 1 Proceedings of the Annual Conference of the International Speech Communication Association INTERSPEECH September 2021 Brno Czechia 606\u2013610."},{"key":"e_1_2_10_16_2","doi-asserted-by":"publisher","DOI":"10.3390\/s22165941"},{"key":"e_1_2_10_17_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11760-022-02156-9"},{"key":"e_1_2_10_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/taffc.2022.3146325"},{"key":"e_1_2_10_19_2","doi-asserted-by":"crossref","unstructured":"WangS. HematiH. GuenasonJ. andBorthD. Generative data augmentation guided by triplet loss for speech emotion recognition 2022 https:\/\/arxiv.org\/abs\/2208.04994.","DOI":"10.21437\/Interspeech.2022-10667"},{"key":"e_1_2_10_20_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2022.108659"},{"key":"e_1_2_10_21_2","doi-asserted-by":"publisher","DOI":"10.1002\/int.22291"},{"key":"e_1_2_10_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/tcsvt.2022.3163445"},{"key":"e_1_2_10_23_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.specom.2022.07.004"},{"key":"e_1_2_10_24_2","unstructured":"KumarP. KaushikV. andRamanB. Towards the explainability of multimodal speech emotion recognition 4 Proceedings of the Annual Conference of the International Speech Communication Association September 2021 Brno Czechia 2927\u20132931."},{"key":"e_1_2_10_25_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cie.2022.108078"},{"key":"e_1_2_10_26_2","volume-title":"Speech Emotion Recognition System Based On Wavelet Transform And Multi-Task Learning","author":"Kexin Z.","year":"2022"},{"key":"e_1_2_10_27_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11042-020-10119-w"},{"key":"e_1_2_10_28_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11265-020-01538-x"},{"key":"e_1_2_10_29_2","unstructured":"LatifS. RanaR. KhalifaS. JurdakR. andSchullerB. W. Multitask learning from augmented auxiliary data for improving speech emotion recognition 2022 https:\/\/arxiv.org\/abs\/2207.05298."},{"key":"e_1_2_10_30_2","doi-asserted-by":"crossref","unstructured":"KangZ. PengJ. WangJ. andXiaoJ. SpeechEQ: Speech emotion recognition based on multi-scale unified datasets and multitask learning 2022 https:\/\/arxiv.org\/abs\/2206.13101.","DOI":"10.21437\/Interspeech.2022-11456"},{"key":"e_1_2_10_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/taslp.2021.3133195"},{"key":"e_1_2_10_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/taffc.2019.2947464"},{"key":"e_1_2_10_33_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00521-022-07410-2"},{"key":"e_1_2_10_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/taslp.2022.3171965"},{"key":"e_1_2_10_35_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2020.106547"},{"key":"e_1_2_10_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/tcds.2020.2990928"},{"key":"e_1_2_10_37_2","doi-asserted-by":"crossref","unstructured":"BarkurR. Deepanshi SureshD. Mahesh KumarT. N. andNarasimhadhanA. V. EnsembleWave: an ensembled approach for automatic speech emotion recognition Proceedings of the IEEE International Conference on Electronics Computing and Communication Technologies July 2022 Bangalore India.","DOI":"10.1109\/CONECCT55679.2022.9865696"},{"key":"e_1_2_10_38_2","doi-asserted-by":"crossref","unstructured":"AlaparthiV. S. PasamT. R. InagandlaD. A. PrakashJ. andSinghP. K. ScSer: supervised contrastive learning for speech emotion recognition using transformers Proceedings of the International Conference on Human System Interaction HSI July 2022 Melbourne Australia.","DOI":"10.1109\/HSI55341.2022.9869453"},{"key":"e_1_2_10_39_2","doi-asserted-by":"crossref","unstructured":"HuangJ. LiY. TaoJ. andLianZ. Speech emotion recognition from variable-length inputs with triplet loss function Proceedings of the Annual Conference of the International Speech Communication Association (InterSpeech) September 2018 Hyderabad India 3673\u20133677.","DOI":"10.21437\/Interspeech.2018-1432"},{"key":"e_1_2_10_40_2","doi-asserted-by":"crossref","unstructured":"DaiD. WuZ. LiR. WuX. JiaJ. andMengH. Learning discriminative features from spectrograms using center loss for speech emotion recognition Proceedings of the IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP) September 2019 Hyderabad India 7405\u20137409.","DOI":"10.1109\/ICASSP.2019.8683765"},{"key":"e_1_2_10_41_2","doi-asserted-by":"crossref","unstructured":"WenY. ZhangK. LiZ. andQiaoY. A discriminative feature learning approach for deep face recognition Proceedings of the European Conference on Computer Vision (ECCV) November 2016 Berlin Germany Springer 499\u2013515.","DOI":"10.1007\/978-3-319-46478-7_31"},{"key":"e_1_2_10_42_2","doi-asserted-by":"crossref","unstructured":"SchroffF. KalenichenkoD. andPhilbinJ. Facenet: a unified embedding for face recognition and clustering Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition June 2015 Boston MA USA 815\u2013823.","DOI":"10.1109\/CVPR.2015.7298682"},{"key":"e_1_2_10_43_2","doi-asserted-by":"crossref","unstructured":"LiZ. HeL. LiJ. WangL. andZhangW.-Q. Towards discriminative representations and unbiased predictions: class-specific angular softmax for speech emotion recognition Proceedings of the Annual Conference of the International Speech Communication Association (InterSpeech) September 2019 Graz Austria 1696\u20131700.","DOI":"10.21437\/Interspeech.2019-1683"},{"key":"e_1_2_10_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/tpami.2021.3087709"},{"key":"e_1_2_10_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/tpami.2018.2858826"},{"key":"e_1_2_10_46_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33018577"},{"key":"e_1_2_10_47_2","doi-asserted-by":"publisher","DOI":"10.18280\/ts.370308"},{"key":"e_1_2_10_48_2","doi-asserted-by":"crossref","unstructured":"GorrostietaC. LotfianR. andTaylorK. Gender de-biasing in speech emotion recognition Proceedings of the Annual Conference of the International Speech Communication Association (InterSpeech) September 2019 Graz Austria 2823\u20132827.","DOI":"10.21437\/Interspeech.2019-1708"},{"key":"e_1_2_10_49_2","doi-asserted-by":"publisher","DOI":"10.4018\/ijiit.2019100102"},{"key":"e_1_2_10_50_2","doi-asserted-by":"crossref","unstructured":"KhareA. ParthasarathyS. andSundaramS. Multi-Modal embeddings using multi-task learning for emotion recognition Proceedings of the Interspeech October 2020 384\u2013388.","DOI":"10.21437\/Interspeech.2020-1827"},{"key":"e_1_2_10_51_2","doi-asserted-by":"crossref","unstructured":"WangF. XiangX. ChengJ. andYuilleA. L. Normface: L2 hypersphere embedding for face verification Proceedings of the ACM International Conference on Multimedia October 2017 Mountain View CA USA 1041\u20131049.","DOI":"10.1145\/3123266.3123359"},{"key":"e_1_2_10_52_2","unstructured":"AbadiM. AgarwalA. BarhamP. BrevdoE. ChenZ. CitroC. CorradoG. S. DavisA. DeanJ. andDevinM. Tensorflow: large-scale machine learning on heterogeneous distributed systems 2016 https:\/\/arxiv.org\/abs\/1603.04467."},{"key":"e_1_2_10_53_2","unstructured":"HaqS. JacksonP. J. B. andEdgeJ. Audio-visual feature selection and reduction for emotion classification Proceedings of the International Conference on Auditory-Visual Speech Processing September 2008 Australia 185\u2013190."},{"key":"e_1_2_10_54_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10579-008-9076-6"},{"key":"e_1_2_10_55_2","unstructured":"HaqS.andJacksonP. J. B. Speaker-dependent audio-visual emotion recognition Proceedings of the International Conference on Audio-Visual Speech Processing October 2009 Norwich UK 53\u201358."},{"key":"e_1_2_10_56_2","unstructured":"BurkhardtF. PaeschkeA. RolfesM. SendlmeierW. andWeissB. A database of German emotional speech Proceedings of the 9th European Conference on Speech Communication and Technology September 2005 Lisbon Portugal 1517\u20131520."},{"key":"e_1_2_10_57_2","doi-asserted-by":"crossref","unstructured":"YenigallaP. KumarA. TripathiS. SinghC. KarS. andVepaJ. Speech emotion recognition using spectrogram and phoneme embedding Proceedings of the Interspeech August 2018 Hyderabad India 3688\u20133692.","DOI":"10.21437\/Interspeech.2018-1811"},{"key":"e_1_2_10_58_2","doi-asserted-by":"crossref","unstructured":"SattA. RozenbergS. andHooryR. Efficient emotion recognition from speech using deep learning on spectrograms Proceedings of the 18th Annual Conference of the International Speech Communication Association August 2017 Stockholm Sweden 1089\u20131093.","DOI":"10.21437\/Interspeech.2017-200"},{"key":"e_1_2_10_59_2","doi-asserted-by":"crossref","unstructured":"BhosaleS. ChakrabortyR. andKopparapuS. K. Deep encoded linguistic and acoustic cues for attention based end to end speech emotion recognition Proceedings of the ICASSP 2020-2020 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP) May 2020 Barcelona Spain 7189\u20137193.","DOI":"10.1109\/ICASSP40776.2020.9054621"},{"key":"e_1_2_10_60_2","doi-asserted-by":"crossref","unstructured":"HuangG.andLiuZ. Densely connected convolutional networks Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) July 2017 Honolulu HI USA 1\u20134.","DOI":"10.1109\/CVPR.2017.243"},{"key":"e_1_2_10_61_2","unstructured":"JiaD. DongW. SocherR. JiaL. LiK. andFeiL. Imagenet: a large-scale hierarchical image database Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition June 2009 Miami FL USA 248\u2013255."},{"key":"e_1_2_10_62_2","first-page":"2579","article-title":"Visualizing data using t-SNE","volume":"9","author":"Van der Maaten L.","year":"2008","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_10_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/taslp.2021.3120586"},{"key":"e_1_2_10_64_2","doi-asserted-by":"crossref","unstructured":"MaX. WuZ. JiaJ. XuM. MengH. andCaiL. Emotion recognition from variable-length speech segments using deep learning on spectrograms Proceedings of the Annual Conference of the International Speech Communication Association (InterSpeech) November 2018 Hyderabad India 3683\u20133687.","DOI":"10.21437\/Interspeech.2018-2228"},{"key":"e_1_2_10_65_2","doi-asserted-by":"crossref","unstructured":"FengH. UenoS. andKawaharaT. End-to-End speech emotion recognition combined with acoustic-to-word ASR model Proceedings of the Interspeech October 2020 Organization of Virtual Conference Sessions 501\u2013505.","DOI":"10.21437\/Interspeech.2020-1180"}],"container-title":["International Journal of Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/downloads.hindawi.com\/journals\/ijis\/2023\/3662839.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/downloads.hindawi.com\/journals\/ijis\/2023\/3662839.xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1155\/2023\/3662839","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,31]],"date-time":"2024-12-31T05:22:54Z","timestamp":1735622574000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1155\/2023\/3662839"}},"subtitle":[],"editor":[{"given":"Mohammad R.","family":"Khosravi","sequence":"additional","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2023,1]]},"references-count":65,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,1]]}},"alternative-id":["10.1155\/2023\/3662839"],"URL":"https:\/\/doi.org\/10.1155\/2023\/3662839","archive":["Portico"],"relation":{},"ISSN":["0884-8173","1098-111X"],"issn-type":[{"value":"0884-8173","type":"print"},{"value":"1098-111X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1]]},"assertion":[{"value":"2022-11-21","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-10-03","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-10-17","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"3662839"}}