{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,17]],"date-time":"2026-06-17T16:46:17Z","timestamp":1781714777842,"version":"3.54.5"},"reference-count":43,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2024,3,7]],"date-time":"2024-03-07T00:00:00Z","timestamp":1709769600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,3,7]],"date-time":"2024-03-07T00:00:00Z","timestamp":1709769600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Guangxi Universities Young and Middle-aged Teachers Basic Ability Improvement Project","award":["2022KY0050"],"award-info":[{"award-number":["2022KY0050"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62307009"],"award-info":[{"award-number":["62307009"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62167001"],"award-info":[{"award-number":["62167001"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62106054"],"award-info":[{"award-number":["62106054"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Neural Process Lett"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Facial emotion recognition in the wild is an important task in computer vision, but it still remains challenging since the influence of backgrounds, occlusions and illumination variations in facial images, as well as the ambiguity of expressions. This paper proposes a harmonious mutual learning framework for emotion recognition, mainly through utilizing attention mechanisms and probability distributions without utilizing additional information. Specifically, this paper builds an architecture with two emotion recognition networks and makes progressive cooperation and interaction between them. We first integrate self-mutual attention module into the backbone to learn discriminative features against the influence from emotion-irrelevant facial information. In this process, we deploy spatial attention module and convolutional block attention module for the two networks respectively, guiding to enhanced and supplementary learning of attention. Further, in the classification head, we propose to learn the latent ground-truth emotion probability distributions using softmax function with temperature to characterize the expression ambiguity. On this basis, a probability distribution distillation learning module is constructed to perform class semantic interaction using bi-directional KL loss, allowing mutual calibration for the two networks. Experimental results on three public datasets show the superiority of the proposed method compared to state-of-the-art ones.<\/jats:p>","DOI":"10.1007\/s11063-024-11566-4","type":"journal-article","created":{"date-parts":[[2024,3,7]],"date-time":"2024-03-07T21:12:48Z","timestamp":1709845968000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Harmonious Mutual Learning for Facial Emotion Recognition"],"prefix":"10.1007","volume":"56","author":[{"given":"Yanling","family":"Gan","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Luhui","family":"Xu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Haiying","family":"Xia","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gan","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,3,7]]},"reference":[{"key":"11566_CR1","doi-asserted-by":"crossref","unstructured":"Wilhelm T (2019) Towards facial expression analysis in a driver assistance system. In: 2019 14th IEEE international conference on automatic face and gesture recognition (FG 2019), pp 1\u2013 4 . IEEE","DOI":"10.1109\/FG.2019.8756565"},{"issue":"4","key":"11566_CR2","doi-asserted-by":"publisher","first-page":"1611","DOI":"10.1007\/s11036-019-01366-9","volume":"25","author":"M Sajjad","year":"2020","unstructured":"Sajjad M, Zahir S, Ullah A, Akhtar Z, Muhammad K (2020) Human behavior understanding in big multimedia data using cnn based facial expression recognition. Mobile Netw Appl 25(4):1611\u20131621","journal-title":"Mobile Netw Appl"},{"issue":"4","key":"11566_CR3","doi-asserted-by":"publisher","first-page":"2132","DOI":"10.1109\/TAFFC.2022.3188390","volume":"13","author":"AV Savchenko","year":"2022","unstructured":"Savchenko AV, Savchenko LV, Makarov I (2022) Classifying emotions and engagement in online learning based on a single facial expression recognition neural network. IEEE Trans Affect Comput 13(4):2132\u20132143","journal-title":"IEEE Trans Affect Comput"},{"key":"11566_CR4","doi-asserted-by":"crossref","unstructured":"Li B, Mehta S, Aneja D, Foster C, Ventola P, Shic F, Shapiro L ( 2019) A facial affect analysis system for autism spectrum disorder. In: 2019 IEEE international conference on image processing (ICIP), pp 4549\u2013 4553 . IEEE","DOI":"10.1109\/ICIP.2019.8803604"},{"key":"11566_CR5","doi-asserted-by":"publisher","first-page":"7383","DOI":"10.1109\/ACCESS.2020.2963913","volume":"8","author":"Y Gan","year":"2020","unstructured":"Gan Y, Chen J, Yang Z, Xu L (2020) Multiple attention network for facial expression recognition. IEEE Access 8:7383\u20137393","journal-title":"IEEE Access"},{"issue":"2","key":"11566_CR6","doi-asserted-by":"publisher","first-page":"199","DOI":"10.3390\/biomimetics8020199","volume":"8","author":"Z Wen","year":"2023","unstructured":"Wen Z, Lin W, Wang T, Xu G (2023) Distract your attention: multi-head cross attention network for facial expression recognition. Biomimetics 8(2):199","journal-title":"Biomimetics"},{"key":"11566_CR7","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2023.109496","volume":"139","author":"Y Liu","year":"2023","unstructured":"Liu Y, Peng J, Dai W, Zeng J, Shan S (2023) Joint spatial and scale attention network for multi-view facial expression recognition. Pattern Recognit. 139:109496","journal-title":"Pattern Recognit."},{"issue":"7","key":"11566_CR8","doi-asserted-by":"publisher","first-page":"9543","DOI":"10.1007\/s12652-023-04627-4","volume":"14","author":"S Indolia","year":"2023","unstructured":"Indolia S, Nigam S, Singh R (2023) A framework for facial expression recognition using deep self-attention network. J Ambient Intell Human Comput 14(7):9543\u20139562","journal-title":"J Ambient Intell Human Comput"},{"key":"11566_CR9","doi-asserted-by":"crossref","unstructured":"Chen S, Wang J, Chen Y, Shi Z, Geng X, Rui Y (2020) Label distribution learning on auxiliary label space graphs for facial expression recognition. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, pp 13984\u2013 13993","DOI":"10.1109\/CVPR42600.2020.01400"},{"key":"11566_CR10","doi-asserted-by":"crossref","unstructured":"Le N, Nguyen K, Tran Q, Tjiputra E, Le B, Nguyen A (2023) Uncertainty-aware label distribution learning for facial expression recognition. In: Proceedings of the IEEE\/CVF winter conference on applications of computer vision, pp 6088\u2013 6097","DOI":"10.1109\/WACV56688.2023.00603"},{"key":"11566_CR11","first-page":"113","volume":"36","author":"X Bian","year":"2022","unstructured":"Bian X, Qin B, Xin X, Li J, Su X, Wang Y (2022) Handwritten mathematical expression recognition via attention aggregation based bi-directional mutual learning. Proc the AAAI Conf Artif Intell 36:113\u2013121","journal-title":"Proc the AAAI Conf Artif Intell"},{"key":"11566_CR12","doi-asserted-by":"crossref","unstructured":"Qiao Y, Jing L, Song X, Chen X, Zhu L, Nie L (2023) Mutual-enhanced incongruity learning network for multi-modal sarcasm detection. In: Proceedings of the AAAI conference on artificial intelligence, vol 37, pp 9507\u2013 9515","DOI":"10.1609\/aaai.v37i8.26138"},{"issue":"12","key":"11566_CR13","doi-asserted-by":"publisher","first-page":"2127","DOI":"10.1109\/TASLP.2019.2942160","volume":"27","author":"Q Ma","year":"2019","unstructured":"Ma Q, Yu L, Tian S, Chen E, Ng WW (2019) Global-local mutual attention model for text classification. IEEE\/ACM Trans Audio Speech Lang Process 27(12):2127\u20132139","journal-title":"IEEE\/ACM Trans Audio Speech Lang Process"},{"issue":"8","key":"11566_CR14","doi-asserted-by":"publisher","first-page":"5361","DOI":"10.1109\/TCSVT.2022.3144775","volume":"32","author":"D Zhang","year":"2022","unstructured":"Zhang D, Zhang Z, Ju Y, Wang C, Xie Y, Qu Y (2022) Dual mutual learning for cross-modality person re-identification. IEEE Trans Circuits Syst Video Technol 32(8):5361\u20135373","journal-title":"IEEE Trans Circuits Syst Video Technol"},{"key":"11566_CR15","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2023.109550","volume":"140","author":"D Liu","year":"2023","unstructured":"Liu D, Zhao L, Wang Y, Kato J (2023) Learn from each other to classify better: cross-layer mutual attention learning for fine-grained visual classification. Pattern Recognit 140:109550","journal-title":"Pattern Recognit"},{"key":"11566_CR16","doi-asserted-by":"crossref","unstructured":"Meng Z, Liu P, Cai J, Han S, Tong Y ( 2017) Identity-aware convolutional neural network for facial expression recognition. In: 2017 12th IEEE international conference on automatic face and gesture recognition (FG 2017), pp 558\u2013 565 . IEEE","DOI":"10.1109\/FG.2017.140"},{"key":"11566_CR17","doi-asserted-by":"crossref","unstructured":"Zhao X, Liang X, Liu L, Li T, Han Y, Vasconcelos N, Yan S ( 2016) Peak-piloted deep network for facial expression recognition. In: Proceedings of the European conference on computer vision (ECCV), pp 425\u2013 442. Springer","DOI":"10.1007\/978-3-319-46475-6_27"},{"issue":"9","key":"11566_CR18","doi-asserted-by":"publisher","first-page":"4193","DOI":"10.1109\/TIP.2017.2689999","volume":"26","author":"K Zhang","year":"2017","unstructured":"Zhang K, Huang Y, Du Y, Wang L (2017) Facial expression recognition based on deep evolutional spatial-temporal networks. IEEE Trans Image Process 26(9):4193\u20134203","journal-title":"IEEE Trans Image Process"},{"key":"11566_CR19","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.patcog.2018.11.001","volume":"88","author":"X Liu","year":"2019","unstructured":"Liu X, Kumar BV, Jia P, You J (2019) Hard negative generation for identity-disentangled facial expression recognition. Pattern Recognit 88:1\u201312","journal-title":"Pattern Recognit"},{"key":"11566_CR20","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1016\/j.ins.2021.07.034","volume":"578","author":"Y Liu","year":"2021","unstructured":"Liu Y, Dai W, Fang F, Chen Y, Huang R, Wang R, Wan B (2021) Dynamic multi-channel metric network for joint pose-aware and identity-invariant facial expression recognition. Inf Sci 578:195\u2013213","journal-title":"Inf Sci"},{"key":"11566_CR21","doi-asserted-by":"crossref","unstructured":"Zhang Y, Xiang T, Hospedales TM, Lu H (2018) Deep mutual learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4320\u2013 4328","DOI":"10.1109\/CVPR.2018.00454"},{"key":"11566_CR22","doi-asserted-by":"crossref","unstructured":"Xu Y, Kan M, Shan S, Chen X (2022) Mutual learning of joint and separate domain alignments for multi-source domain adaptation. In: Proceedings of the IEEE\/CVF winter conference on applications of computer vision, pp 1890\u2013 1899","DOI":"10.1109\/WACV51458.2022.00172"},{"issue":"4","key":"11566_CR23","doi-asserted-by":"publisher","first-page":"979","DOI":"10.1049\/ipr2.12688","volume":"17","author":"Y Wang","year":"2023","unstructured":"Wang Y, Wang L, Zhou Y (2023) Bi-level deep mutual learning assisted multi-task network for occluded person re-identification. IET Image Process 17(4):979\u2013987","journal-title":"IET Image Process"},{"key":"11566_CR24","doi-asserted-by":"crossref","unstructured":"Woo S, Park J, Lee J-Y, Kweon IS (2018) Cbam: convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV), pp 3\u2013 19","DOI":"10.1007\/978-3-030-01234-2_1"},{"key":"11566_CR25","doi-asserted-by":"crossref","unstructured":"Li S, Deng W, Du J ( 2017) Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2852\u20132861","DOI":"10.1109\/CVPR.2017.277"},{"key":"11566_CR26","doi-asserted-by":"crossref","unstructured":"Goodfellow IJ, Erhan D, Carrier PL, Courville A, Mirza M, Hamner B, Cukierski W, Tang Y, Thaler D, Lee D-H, et al (2013) Challenges in representation learning: a report on three machine learning contests. In: International conference on neural information processing, pp 117\u2013 124. Springer","DOI":"10.1007\/978-3-642-42051-1_16"},{"key":"11566_CR27","doi-asserted-by":"crossref","unstructured":"Dhall A, Ramana\u00a0Murthy O, Goecke R, Joshi J, Gedeon T (2015) Video and image based emotion recognition challenges in the wild: emotiw 2015. In: Proceedings of the 2015 ACM on international conference on multimodal interaction, pp. 423\u2013 426","DOI":"10.1145\/2818346.2829994"},{"issue":"10","key":"11566_CR28","doi-asserted-by":"publisher","first-page":"1499","DOI":"10.1109\/LSP.2016.2603342","volume":"23","author":"K Zhang","year":"2016","unstructured":"Zhang K, Zhang Z, Li Z, Qiao Y (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process Lett 23(10):1499\u20131503","journal-title":"IEEE Signal Process Lett"},{"key":"11566_CR29","doi-asserted-by":"crossref","unstructured":"Guo Y, Zhang L, Hu Y, He X, Gao J (2016) Ms-celeb-1m: a dataset and benchmark for large-scale face recognition. In: Proceedings of the European conference on computer vision (ECCV), pp 87\u2013 102. Springer","DOI":"10.1007\/978-3-319-46487-9_6"},{"key":"11566_CR30","doi-asserted-by":"crossref","unstructured":"Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, pp 248\u2013255 . IEEE","DOI":"10.1109\/CVPR.2009.5206848"},{"issue":"5","key":"11566_CR31","doi-asserted-by":"publisher","first-page":"2439","DOI":"10.1109\/TIP.2018.2886767","volume":"28","author":"Y Li","year":"2018","unstructured":"Li Y, Zeng J, Shan S, Chen X (2018) Occlusion aware facial expression recognition using cnn with attention mechanism. IEEE Trans Image Process 28(5):2439\u20132450","journal-title":"IEEE Trans Image Process"},{"key":"11566_CR32","doi-asserted-by":"publisher","first-page":"4057","DOI":"10.1109\/TIP.2019.2956143","volume":"29","author":"K Wang","year":"2020","unstructured":"Wang K, Peng X, Yang J, Meng D, Qiao Y (2020) Region attention networks for pose and occlusion robust facial expression recognition. IEEE Trans Image Process 29:4057\u20134069","journal-title":"IEEE Trans Image Process"},{"key":"11566_CR33","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1016\/j.neucom.2020.12.076","volume":"432","author":"H Li","year":"2021","unstructured":"Li H, Wang N, Yu Y, Yang X, Gao X (2021) Lban-il: a novel method of high discriminative representation for facial expression recognition. Neurocomputing 432:159\u2013169","journal-title":"Neurocomputing"},{"issue":"3","key":"11566_CR34","doi-asserted-by":"publisher","first-page":"1927","DOI":"10.1109\/TAFFC.2022.3156920","volume":"14","author":"J Cai","year":"2023","unstructured":"Cai J, Meng Z, Khan AS, Li Z, O\u2019Reilly J, Tong Y (2023) Probabilistic attribute tree structured convolutional neural networks for facial expression recognition in the wild. IEEE Trans Affect Comput 14(3):1927\u20131941","journal-title":"IEEE Trans Affect Comput"},{"issue":"10","key":"11566_CR35","doi-asserted-by":"publisher","first-page":"5001","DOI":"10.1007\/s00371-022-02642-8","volume":"39","author":"Y Xi","year":"2023","unstructured":"Xi Y, Mao Q, Zhou L (2023) Weighted contrastive learning using pseudo labels for facial expression recognition. Vis Comput 39(10):5001\u20135012","journal-title":"Vis Comput"},{"key":"11566_CR36","doi-asserted-by":"publisher","first-page":"105","DOI":"10.1016\/j.patrec.2019.04.002","volume":"125","author":"Y Gan","year":"2019","unstructured":"Gan Y, Chen J, Xu L (2019) Facial expression recognition boosted by soft label with a diverse ensemble. Pattern Recognit Lett 125:105\u2013112","journal-title":"Pattern Recognit Lett"},{"key":"11566_CR37","doi-asserted-by":"crossref","unstructured":"Ruan D, Yan Y, Lai S, Chai Z, Shen C, Wang H ( 2021) Feature decomposition and reconstruction learning for effective facial expression recognition. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, pp 7660\u20137669","DOI":"10.1109\/CVPR46437.2021.00757"},{"issue":"9","key":"11566_CR38","doi-asserted-by":"publisher","first-page":"3046","DOI":"10.3390\/s21093046","volume":"21","author":"S Minaee","year":"2021","unstructured":"Minaee S, Minaei M, Abdolrashidi A (2021) Deep-emotion: facial expression recognition using attentional convolutional network. Sensors 21(9):3046","journal-title":"Sensors"},{"issue":"12","key":"11566_CR39","doi-asserted-by":"publisher","first-page":"15251","DOI":"10.1007\/s11042-017-5105-z","volume":"77","author":"D Li","year":"2018","unstructured":"Li D, Wen G (2018) Mrmr-based ensemble pruning for facial expression recognition. Multimedia Tools Appl 77(12):15251\u201315272","journal-title":"Multimedia Tools Appl"},{"issue":"5","key":"11566_CR40","doi-asserted-by":"publisher","first-page":"597","DOI":"10.1007\/s12559-017-9472-6","volume":"9","author":"G Wen","year":"2017","unstructured":"Wen G, Hou Z, Li H, Li D, Jiang L, Xun E (2017) Ensemble of deep neural networks with probability-based fusion for facial expression recognition. Cognit Comput 9(5):597\u2013610","journal-title":"Cognit Comput"},{"key":"11566_CR41","doi-asserted-by":"crossref","unstructured":"Xiang J, Zhu G ( 2017) Joint face detection and facial expression recognition with mtcnn. In: 2017 4th International conference on information science and control engineering (ICISCE), pp 424\u2013 427 . IEEE","DOI":"10.1109\/ICISCE.2017.95"},{"key":"11566_CR42","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1016\/j.neucom.2019.05.005","volume":"355","author":"J Shao","year":"2019","unstructured":"Shao J, Qian Y (2019) Three convolutional neural network models for facial expression recognition in the wild. Neurocomputing 355:82\u201392","journal-title":"Neurocomputing"},{"key":"11566_CR43","doi-asserted-by":"crossref","unstructured":"Liu X, Vijaya\u00a0Kumar B, You J, Jia P (2017) Adaptive deep metric learning for identity-aware facial expression recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 20\u201329","DOI":"10.1109\/CVPRW.2017.79"}],"container-title":["Neural Processing Letters"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11063-024-11566-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11063-024-11566-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11063-024-11566-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,16]],"date-time":"2024-05-16T20:34:28Z","timestamp":1715891668000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11063-024-11566-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,7]]},"references-count":43,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2024,4]]}},"alternative-id":["11566"],"URL":"https:\/\/doi.org\/10.1007\/s11063-024-11566-4","relation":{},"ISSN":["1573-773X"],"issn-type":[{"value":"1573-773X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,7]]},"assertion":[{"value":"11 February 2024","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 March 2024","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}],"article-number":"96"}}