{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,9]],"date-time":"2026-05-09T17:09:27Z","timestamp":1778346567655,"version":"3.51.4"},"reference-count":41,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,8,17]],"date-time":"2023-08-17T00:00:00Z","timestamp":1692230400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:p>Recognizing occluded facial expressions in the wild poses a significant challenge. However, most previous approaches rely solely on either global or local feature-based methods, leading to the loss of relevant expression features. To address these issues, a feature fusion residual attention network (FFRA-Net) is proposed. FFRA-Net consists of a multi-scale module, a local attention module, and a feature fusion module. The multi-scale module divides the intermediate feature map into several sub-feature maps in an equal manner along the channel dimension. Then, a convolution operation is applied to each of these feature maps to obtain diverse global features. The local attention module divides the intermediate feature map into several sub-feature maps along the spatial dimension. Subsequently, a convolution operation is applied to each of these feature maps, resulting in the extraction of local key features through the attention mechanism. The feature fusion module plays a crucial role in integrating global and local expression features while also establishing residual links between inputs and outputs to compensate for the loss of fine-grained features. Last, two occlusion expression datasets (FM_RAF-DB and SG_RAF-DB) were constructed based on the RAF-DB dataset. Extensive experiments demonstrate that the proposed FFRA-Net achieves excellent results on four datasets: FM_RAF-DB, SG_RAF-DB, RAF-DB, and FERPLUS, with accuracies of 77.87%, 79.50%, 88.66%, and 88.97%, respectively. Thus, the approach presented in this paper demonstrates strong applicability in the context of occluded facial expression recognition (FER).<\/jats:p>","DOI":"10.3389\/fnbot.2023.1250706","type":"journal-article","created":{"date-parts":[[2023,8,18]],"date-time":"2023-08-18T14:13:03Z","timestamp":1692367983000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["Occlusion facial expression recognition based on feature fusion residual attention network"],"prefix":"10.3389","volume":"17","author":[{"given":"Yuekun","family":"Chen","sequence":"first","affiliation":[]},{"given":"Shuaishi","family":"Liu","sequence":"additional","affiliation":[]},{"given":"Dongxu","family":"Zhao","sequence":"additional","affiliation":[]},{"given":"Wenkai","family":"Ji","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2023,8,17]]},"reference":[{"key":"B1","doi-asserted-by":"crossref","first-page":"292","DOI":"10.1145\/3240508.3240578","article-title":"\u201cEmotion recognition in speech using cross-modal transfer in the wild,\u201d","volume-title":"Proceedings of the 26th ACM international conference on Multimedia, pages","author":"Albanie","year":"2018"},{"key":"B2","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1145\/2993148.2993165","article-title":"\u201cTraining deep networks for facial expression recognition with crowd-sourced label distribution,\u201d","volume-title":"Proceedings of the 18th ACM International Conference on Multimodal Interaction","author":"Barsoum","year":"2016"},{"key":"B3","doi-asserted-by":"crossref","first-page":"14960","DOI":"10.1109\/ICCV48922.2021.01471","article-title":"\u201cUnderstanding and mitigating annotation bias in facial expression recognition,\u201d","volume-title":"2021 IEEE\/CVF International Conference on Computer Vision (ICCV)","author":"Chen","year":"2021"},{"key":"B4","first-page":"1","article-title":"\u201cFacial motion prior networks for facial expression recognition,\u201d","volume-title":"2019 IEEE Visual Communications and Image Processing (VCIP)","author":"Chen","year":"2019"},{"key":"B5","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR42600.2020.00525","article-title":"\u201cRetinaface: Single-shot multi-level face localisation in the wild,\u201d","volume-title":"2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Deng","year":"2020"},{"key":"B6","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/IJCB48548.2020.9304923","article-title":"\u201cOcclusion-adaptive deep network for robust facial expression recognition,\u201d","volume-title":"2020 IEEE International Joint Conference on Biometrics (IJCB)","author":"Ding","year":"2020"},{"key":"B7","doi-asserted-by":"crossref","first-page":"2401","DOI":"10.1109\/WACV48630.2021.00245","article-title":"Facial expression recognition in the wild via deep attentive center loss,\u201d","volume-title":"2021 IEEE Winter Conference on Applications of Computer Vision (WACV)","author":"Farzaneh","year":"2021"},{"key":"B8","doi-asserted-by":"publisher","first-page":"652","DOI":"10.1109\/TPAMI.2019.2938758","article-title":"Res2net: A new multi-scale backbone architecture","volume":"43","author":"Gao","year":"2019","journal-title":"IEEE Trans. Patt. Analy. Mach. Intell"},{"key":"B9","first-page":"770","article-title":"\u201cDeep residual learning for image recognition,\u201d","volume-title":"2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","author":"He","year":"2015"},{"key":"B10","doi-asserted-by":"publisher","first-page":"2026","DOI":"10.3390\/s21062026","article-title":"The extensive usage of the facial image threshing machine for facial emotion recognition performance","volume":"21","author":"Kim","year":"2021","journal-title":"Sensors"},{"key":"B11","doi-asserted-by":"publisher","first-page":"2016","DOI":"10.1109\/TIP.2021.3049955","article-title":"Adaptively learning facial expression representation via c-f labels and distillation","volume":"30","author":"Li","year":"2021","journal-title":"IEEE Trans. Image Proc"},{"key":"B12","doi-asserted-by":"publisher","first-page":"356","DOI":"10.1109\/TIP.2018.2868382","article-title":"Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition","volume":"28","author":"Li","year":"2019","journal-title":"IEEE Trans. Image Proc."},{"key":"B13","doi-asserted-by":"crossref","first-page":"2209","DOI":"10.1109\/ICPR.2018.8545853","article-title":"\u201cPatch-gated cnn for occlusion-aware facial expression recognition,\u201d","volume-title":"2018 24th International Conference on Pattern Recognition (ICPR)","author":"Li","year":"2018"},{"key":"B14","doi-asserted-by":"publisher","first-page":"2439","DOI":"10.1109\/TIP.2018.2886767","article-title":"Occlusion aware facial expression recognition using cnn with attention mechanism","volume":"28","author":"Li","year":"2019","journal-title":"IEEE Trans. Image Proc"},{"key":"B15","first-page":"94","article-title":"\u201cThe extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression,\u201d","volume-title":"2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops","author":"Lucey","year":"2010"},{"key":"B16","doi-asserted-by":"publisher","first-page":"1057983","DOI":"10.3389\/fnbot.2022.1057983","article-title":"Facial expression recognition method based on PSA-YOLO network","volume":"16","author":"Ma","year":"2023","journal-title":"Front. Neurorob"},{"key":"B17","doi-asserted-by":"crossref","first-page":"837","DOI":"10.1109\/CVPRW.2019.00112","article-title":"\u201cFeratt: Facial expression recognition with attention net,\u201d","volume-title":"2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)","author":"Marrero-Fern\u00e1ndez","year":"2019"},{"key":"B18","doi-asserted-by":"crossref","first-page":"566","DOI":"10.1145\/3343031.3351049","article-title":"\u201cOccluded facial expression recognition enhanced through privileged information,\u201d","volume-title":"Proceedings of the 27th ACM International Conference on Multimedia","author":"Pan","year":"2019"},{"key":"B19","first-page":"1072","article-title":"\u201cFeature vector extraction technique for facial emotion recognition using facial landmarks,\u201d","volume-title":"2021 International Conference on Information and Communication Technology Convergence (ICTC)","author":"Poulose","year":""},{"key":"B20","first-page":"356","article-title":"\u201cForeground extraction based facial emotion recognition using deep learning xception model,\u201d","volume-title":"2021 Twelfth International Conference on Ubiquitous and Future Networks (ICUFN)","author":"Poulose","year":""},{"key":"B21","doi-asserted-by":"publisher","first-page":"864490","DOI":"10.3389\/fnins.2022.864490","article-title":"The recognition of facial expressions under surgical masks: The primacy of anger","volume":"16","author":"Proverbio","year":"2022","journal-title":"Front. Neurorob"},{"key":"B22","first-page":"11154","article-title":"\u201cAu-expression knowledge constrained representation learning for facial expression recognition,\u201d","volume-title":"2021 IEEE International Conference on Robotics and Automation (ICRA)","author":"Pu","year":"2020"},{"key":"B23","doi-asserted-by":"crossref","first-page":"4510","DOI":"10.1109\/CVPR.2018.00474","article-title":"\u201cMobilenetv2: Inverted residuals and linear bottlenecks,\u201d","volume-title":"2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Sandler","year":"2018"},{"key":"B24","doi-asserted-by":"crossref","first-page":"6244","DOI":"10.1109\/CVPR46437.2021.00618","article-title":"\u201cDive into ambiguity: Latent distribution mining and pairwise uncertainty estimation for facial expression recognition,\u201d","volume-title":"2021 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","author":"She","year":"2021"},{"key":"B25","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1409.1556","article-title":"Very deep convolutional networks for large-scale image recognition","author":"Simonyan","year":"2014","journal-title":"arXiv preprint arXiv"},{"key":"B26","article-title":"Induced disgust, happiness and surprise : an addition to the mmi facial expression database,\u201d","author":"Valstar","year":"2010","journal-title":"Proceedings of the 3rd International Workshop on EMOTION (satellite of LREC): Corpora for Research on Emotion and Affect"},{"key":"B27","doi-asserted-by":"crossref","first-page":"238","DOI":"10.1145\/3343031.3350872","article-title":"\u201cIdentity- and pose-robust facial expression recognition through adversarial feature learning,\u201d","volume-title":"Proceedings of the 27th ACM International Conference on Multimedia","author":"Wang","year":"2019"},{"key":"B28","doi-asserted-by":"crossref","first-page":"6896","DOI":"10.1109\/CVPR42600.2020.00693","article-title":"\u201cSuppressing uncertainties for large-scale facial expression recognition,\u201d","volume-title":"2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Wang","year":"2020"},{"key":"B29","doi-asserted-by":"publisher","first-page":"4057","DOI":"10.1109\/TIP.2019.2956143","article-title":"Region attention networks for pose and occlusion robust facial expression recognition","volume":"29","author":"Wang","year":"2019","journal-title":"IEEE Trans. Image Proc"},{"key":"B30","first-page":"1","article-title":"\u201cFacial expression recognition with deep learning,\u201d","volume-title":"International Conference on Internet Multimedia Computing and Service","author":"Wang","year":"2018"},{"key":"B31","doi-asserted-by":"publisher","first-page":"199","DOI":"10.3390\/biomimetics8020199","article-title":"Distract your attention: Multi-head cross attention network for facial expression recognition","volume":"8","author":"Wen","year":"2021","journal-title":"Biomimetics"},{"key":"B32","doi-asserted-by":"crossref","first-page":"2927","DOI":"10.1145\/3394171.3413773","article-title":"\u201cOccluded facial expression recognition with step-wise assistance from unpaired non-occluded images,\u201d","volume-title":"Proceedings of the 28th ACM International Conference on Multimedia","author":"Xia","year":"2020"},{"key":"B33","doi-asserted-by":"crossref","first-page":"3581","DOI":"10.1109\/ICCV48922.2021.00358","article-title":"\u201cTransfer: Learning relation-aware facial expression representations with transformers,\u201d","volume-title":"2021 IEEE\/CVF International Conference on Computer Vision (ICCV)","author":"Xue","year":"2021"},{"key":"B34","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2022.3226473","article-title":"Vision transformer with attentive pooling for robust facial expression recognition","author":"Xue","year":"2022","journal-title":"IEEE Trans. Affec. Comput"},{"key":"B35","first-page":"227","article-title":"\u201cFacial expression recognition with inconsistently annotated datasets,\u201d","volume-title":"European Conference on Computer Vision","author":"Zeng","year":"2018"},{"key":"B36","doi-asserted-by":"crossref","first-page":"3359","DOI":"10.1109\/CVPR.2018.00354","article-title":"\u201cJoint pose and expression modeling for facial expression recognition,\u201d","volume-title":"2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Zhang","year":"2018"},{"key":"B37","first-page":"2235","article-title":"\u201cSa-net: Shuffle attention for deep convolutional neural networks,\u201d","volume-title":"2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","author":"Zhang","year":"2021"},{"key":"B38","first-page":"17616","article-title":"\u201cRelative uncertainty learning for facial expression recognition,\u201d","volume-title":"Neural Information Processing Systems","author":"Zhang","year":"2021"},{"key":"B39","doi-asserted-by":"publisher","first-page":"607","DOI":"10.1016\/j.imavis.2011.07.002","article-title":"Facial expression recognition from near-infrared videos","volume":"29","author":"Zhao","year":"2011","journal-title":"Image Vision Comput"},{"key":"B40","doi-asserted-by":"publisher","first-page":"6544","DOI":"10.1109\/TIP.2021.3093397","article-title":"Learning deep global multi-scale and local attention features for facial expression recognition in the wild","volume":"30","author":"Zhao","year":"2021","journal-title":"IEEE Trans. Image Proc"},{"key":"B41","first-page":"2921","article-title":"\u201cLearning deep features for discriminative localization,\u201d","volume-title":"2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Zhou","year":"2015"}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1250706\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,18]],"date-time":"2023-08-18T14:13:29Z","timestamp":1692368009000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1250706\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,17]]},"references-count":41,"alternative-id":["10.3389\/fnbot.2023.1250706"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2023.1250706","relation":{},"ISSN":["1662-5218"],"issn-type":[{"value":"1662-5218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,17]]},"article-number":"1250706"}}