{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,29]],"date-time":"2026-05-29T20:09:45Z","timestamp":1780085385997,"version":"3.54.0"},"reference-count":45,"publisher":"MDPI AG","issue":"16","license":[{"start":{"date-parts":[[2024,8,21]],"date-time":"2024-08-21T00:00:00Z","timestamp":1724198400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"General Project for Education of National Social Science Fund","award":["BCA230278"],"award-info":[{"award-number":["BCA230278"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>In recent years, significant progress has been made in facial expression recognition methods. However, tasks related to facial expression recognition in real environments still require further research. This paper proposes a tri-cross-attention transformer with a multi-feature fusion network (TriCAFFNet) to improve facial expression recognition performance under challenging conditions. By combining LBP (Local Binary Pattern) features, HOG (Histogram of Oriented Gradients) features, landmark features, and CNN (convolutional neural network) features from facial images, the model is provided with a rich input to improve its ability to discern subtle differences between images. Additionally, tri-cross-attention blocks are designed to facilitate information exchange between different features, enabling mutual guidance among different features to capture salient attention. Extensive experiments on several widely used datasets show that our TriCAFFNet achieves the SOTA performance on RAF-DB with 92.17%, AffectNet (7 cls) with 67.40%, and AffectNet (8 cls) with 63.49%, respectively.<\/jats:p>","DOI":"10.3390\/s24165391","type":"journal-article","created":{"date-parts":[[2024,8,23]],"date-time":"2024-08-23T12:53:19Z","timestamp":1724417599000},"page":"5391","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["TriCAFFNet: A Tri-Cross-Attention Transformer with a Multi-Feature Fusion Network for Facial Expression Recognition"],"prefix":"10.3390","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9097-4639","authenticated-orcid":false,"given":"Yuan","family":"Tian","sequence":"first","affiliation":[{"name":"Faculty of Artificial Intelligence in Education, Central China Normal University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhao","family":"Wang","sequence":"additional","affiliation":[{"name":"Faculty of Artificial Intelligence in Education, Central China Normal University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-3863-7566","authenticated-orcid":false,"given":"Di","family":"Chen","sequence":"additional","affiliation":[{"name":"Faculty of Artificial Intelligence in Education, Central China Normal University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Huang","family":"Yao","sequence":"additional","affiliation":[{"name":"Faculty of Artificial Intelligence in Education, Central China Normal University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2024,8,21]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1357","DOI":"10.1109\/34.817413","article-title":"Automatic classification of single facial images","volume":"21","author":"Lyons","year":"1999","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., and Matthews, I. (2010, January 13\u201318). The extended cohn-kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, San Francisco, CA, USA.","DOI":"10.1109\/CVPRW.2010.5543262"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Li, S., Deng, W., and Du, J. (2017, January 21\u201326). Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.277"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1109\/TAFFC.2017.2740923","article-title":"AffectNet: A database for facial expression, valence, and arousal computing in the wild","volume":"10","author":"Mollahosseini","year":"2019","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Barsoum, E., Zhang, C., Ferrer, C.C., and Zhang, Z. (2016, January 12\u201316). Training deep networks for facial expression recognition with crowd-sourced label distribution. Proceedings of the 18th ACM International Conference on Multimodal Interaction, ICMI \u201916, Tokyo, Japan.","DOI":"10.1145\/2993148.2993165"},{"key":"ref_6","unstructured":"Dalal, N., and Triggs, B. (2005, January 20\u201325). Histograms of oriented gradients for human detection. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR\u201905), San Diego, CA, USA."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"971","DOI":"10.1109\/TPAMI.2002.1017623","article-title":"Multiresolution gray-scale and rotation invariant texture classification with local binary patterns","volume":"24","author":"Ojala","year":"2002","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Jabid, T., Kabir, M.H., and Chae, O. (2010, January 9\u201313). Local directional pattern (LDP) for face recognition. Proceedings of the 2010 Digest of Technical Papers International Conference on Consumer Electronics (ICCE), Las Vegas, NV, USA.","DOI":"10.1109\/ICCE.2010.5418801"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1023\/B:VISI.0000029664.99615.94","article-title":"Distinctive image features from scale-invariant keypoints","volume":"60","author":"Lowe","year":"2004","journal-title":"Int. J. Comput. Vis."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Parra, D., and Camargo, C. (2023, January 24\u201326). Design methodology for single-channel CNN-based FER systems. Proceedings of the 2023 6th International Conference on Information and Computer Technologies (ICICT), Raleigh, NC, USA.","DOI":"10.1109\/ICICT58900.2023.00022"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"012004","DOI":"10.1088\/1742-6596\/2236\/1\/012004","article-title":"Deep learning for facial emotion recognition using custom CNN architecture","volume":"2236","author":"Borgalli","year":"2022","journal-title":"J. Phys. Conf. Ser."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"439","DOI":"10.1007\/s40031-021-00681-8","article-title":"FERNet: A Deep CNN Architecture for Facial Expression Recognition in the Wild","volume":"103","author":"Bodapati","year":"2022","journal-title":"J. Inst. Eng. (India) Ser. B"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Wang, K., Peng, X., Yang, J., Lu, S., and Qiao, Y. (2020, January 13\u201319). Suppressing uncertainties for large-scale facial expression recognition. Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00693"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Xue, F., Wang, Q., and Guo, G. (2021, January 10\u201317). TransFER: Learning relation-aware facial expression representations with transformers. Proceedings of the 2021 IEEE\/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.","DOI":"10.1109\/ICCV48922.2021.00358"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Fan, Y., Wang, H., Zhu, X., Cao, X., Yi, C., Chen, Y., Jia, J., and Lu, X. (2022). FER-PCVT: Facial expression recognition with patch-convolutional vision transformer for stroke patients. Brain Sci., 12.","DOI":"10.3390\/brainsci12121626"},{"key":"ref_16","unstructured":"Li, Y., Wang, M., Gong, M., Lu, Y., and Liu, L. (2023). FER-former: Multi-modal transformer for facial expression recognition. arXiv."},{"key":"ref_17","unstructured":"Li, H., Sui, M., Zhao, F., Zha, Z., and Wu, F. (2021). MVT: Mask vision transformer for facial expression recognition in the wild. arXiv."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Yang, W., Chen, H., Peng, H., Guo, J., and Liu, Z. (2023). Leveraging one-class classification with resnet18 and cbam for enhanced facial expression recognition. SSRN.","DOI":"10.2139\/ssrn.4505113"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1236","DOI":"10.1109\/TAFFC.2021.3122146","article-title":"Facial expression recognition with visual transformers and attentional selective fusion","volume":"14","author":"Ma","year":"2023","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Kim, J.H., Kim, N., and Won, C.S. (2022). Facial expression recognition with swin transformer. arXiv.","DOI":"10.3390\/s22103729"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"765","DOI":"10.1109\/TSMCC.2011.2118750","article-title":"Local binary patterns and its application to facial image analysis: A survey","volume":"41","author":"Huang","year":"2011","journal-title":"IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.)"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"803","DOI":"10.1016\/j.imavis.2008.08.005","article-title":"Facial expression recognition based on Local Binary Patterns: A comprehensive study","volume":"27","author":"Shan","year":"2009","journal-title":"Image Vis. Comput."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Hasani, B., and Mahoor, M.H. (2017, January 21\u201326). Facial expression recognition using enhanced deep 3D convolutional neural networks. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, USA.","DOI":"10.1109\/CVPRW.2017.282"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Jung, H., Lee, S., Yim, J., Park, S., and Kim, J. (2015, January 7\u201313). Joint fine-tuning in deep neural networks for facial expression recognition. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.341"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Zheng, C., Mendieta, M., and Chen, C. (2023, January 2\u20136). POSTER: A pyramid cross-fusion transformer network for facial expression recognition. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Paris, France.","DOI":"10.1109\/ICCVW60793.2023.00339"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Savchenko, A.V. (2021, January 16\u201318). Facial expression and attributes recognition based on multi-task learning of lightweight neural networks. Proceedings of the 2021 IEEE 19th International Symposium on Intelligent Systems and Informatics (SISY), Subotica, Serbia.","DOI":"10.1109\/SISY52375.2021.9582508"},{"key":"ref_27","unstructured":"Tang, Y. (2013). Deep Learning using Linear Support Vector Machines. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"131988","DOI":"10.1109\/ACCESS.2020.3010018","article-title":"Pyramid with super resolution for in-the-wild facial expression recognition","volume":"8","author":"Vo","year":"2020","journal-title":"IEEE Access"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"3244","DOI":"10.1109\/TAFFC.2022.3226473","article-title":"Vision transformer with attentive pooling for robust facial expression recognition","volume":"14","author":"Xue","year":"2022","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"7321175","DOI":"10.1155\/2024\/7321175","article-title":"Learning Cognitive Features as Complementary for Facial Expression Recognition","volume":"2024","author":"Li","year":"2024","journal-title":"Int. J. Intell. Syst."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1109\/TAFFC.2016.2593719","article-title":"Facial Expression Recognition in Video with Multiple Feature Fusion","volume":"9","author":"Chen","year":"2018","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1016\/j.neucom.2019.05.005","article-title":"Three convolutional neural network models for facial expression recognition in the wild","volume":"355","author":"Shao","year":"2019","journal-title":"Neurocomputing"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Li, X., Deng, W., Li, S., and Li, Y. (2023, January 17\u201324). Compound Expression Recognition In-the-Wild with AU-Assisted Meta Multi-Task Learning. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.","DOI":"10.1109\/CVPRW59228.2023.00608"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1007\/s13735-023-00285-6","article-title":"Neural style transfer generative adversarial network (NST-GAN) for facial expression recognition","volume":"12","author":"Khemakhem","year":"2023","journal-title":"Int. J. Multimed. Inf. Retr."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Akhand, M.A.H., Roy, S., Siddique, N., Kamal, M.A.S., and Shimamura, T. (2021). Facial Emotion Recognition Using Transfer Learning in the Deep CNN. Electronics, 10.","DOI":"10.3390\/electronics10091036"},{"key":"ref_36","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2023). Attention Is All You Need. arXiv."},{"key":"ref_37","unstructured":"Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021). An Image is Worth 16 \u00d7 16 Words: Transformers for Image Recognition at Scale. arXiv."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"451","DOI":"10.1109\/TAFFC.2020.3031602","article-title":"Facial Expression Recognition in the Wild Using Multi-Level Features and Attention Mechanisms","volume":"14","author":"Li","year":"2023","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Deng, J., Guo, J., Xue, N., and Zafeiriou, S. (2019, January 15\u201320). ArcFace: Additive Angular Margin Loss for Deep Face Recognition. Proceedings of the 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00482"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Zeng, D., Lin, Z., Yan, X., Liu, Y., Wang, F., and Tang, B. (2022, January 18\u201324). Face2Exp: Combating data biases for facial expression recognition. Proceedings of the 2022 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.01965"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Guo, Y., Zhang, L., Hu, Y., He, X., and Gao, J. (2016). MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition. arXiv.","DOI":"10.1007\/978-3-319-46487-9_6"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"2464","DOI":"10.1109\/TCSS.2023.3305616","article-title":"Quaternion deformable local binary pattern and pose-correction facial decomposition for color facial expression recognition in the wild","volume":"11","author":"Jin","year":"2023","journal-title":"IEEE Trans. Comput. Soc. Syst."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Wang, C., Ling, X., and Deng, W. (2022). Learn from all: Erasing attention consistency for noisy label facial expression recognition. arXiv.","DOI":"10.1007\/978-3-031-19809-0_24"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Shi, J., Zhu, S., and Liang, Z. (2021). Learning to amend facial expression representation via de-albino and affinity. arXiv.","DOI":"10.23919\/CCC55666.2022.9901738"},{"key":"ref_45","first-page":"2579","article-title":"Visualizing data using t-SNE","volume":"9","author":"Maaten","year":"2008","journal-title":"J. Mach. Learn. Res."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/16\/5391\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:40:09Z","timestamp":1760110809000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/16\/5391"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,21]]},"references-count":45,"journal-issue":{"issue":"16","published-online":{"date-parts":[[2024,8]]}},"alternative-id":["s24165391"],"URL":"https:\/\/doi.org\/10.3390\/s24165391","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,21]]}}}