{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,29]],"date-time":"2026-05-29T20:06:04Z","timestamp":1780085164628,"version":"3.54.0"},"reference-count":42,"publisher":"MDPI AG","issue":"15","license":[{"start":{"date-parts":[[2023,7,30]],"date-time":"2023-07-30T00:00:00Z","timestamp":1690675200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Collaborative Innovation Center for Informatization and Balanced Development of K-12 Education by MOE and Hubei Province","award":["xtzd2022-008"],"award-info":[{"award-number":["xtzd2022-008"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Facial expressions help individuals convey their emotions. In recent years, thanks to the development of computer vision technology, facial expression recognition (FER) has become a research hotspot and made remarkable progress. However, human faces in real-world environments are affected by various unfavorable factors, such as facial occlusion and head pose changes, which are seldom encountered in controlled laboratory settings. These factors often lead to a reduction in expression recognition accuracy. Inspired by the recent success of transformers in many computer vision tasks, we propose a model called the fine-tuned channel\u2013spatial attention transformer (FT-CSAT) to improve the accuracy of recognition of FER in the wild. FT-CSAT consists of two crucial components: channel\u2013spatial attention module and fine-tuning module. In the channel\u2013spatial attention module, the feature map is input into the channel attention module and the spatial attention module sequentially. The final output feature map will effectively incorporate both channel information and spatial information. Consequently, the network becomes adept at focusing on relevant and meaningful features associated with facial expressions. To further improve the model\u2019s performance while controlling the number of excessive parameters, we employ a fine-tuning method. Extensive experimental results demonstrate that our FT-CSAT outperforms the state-of-the-art methods on two benchmark datasets: RAF-DB and FERPlus. The achieved recognition accuracy is 88.61% and 89.26%, respectively. Furthermore, to evaluate the robustness of FT-CSAT in the case of facial occlusion and head pose changes, we take tests on Occlusion-RAF-DB and Pose-RAF-DB data sets, and the results also show that the superior recognition performance of the proposed method under such conditions.<\/jats:p>","DOI":"10.3390\/s23156799","type":"journal-article","created":{"date-parts":[[2023,7,31]],"date-time":"2023-07-31T03:30:02Z","timestamp":1690774202000},"page":"6799","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["Facial Expression Recognition Based on Fine-Tuned Channel\u2013Spatial Attention Transformer"],"prefix":"10.3390","volume":"23","author":[{"given":"Huang","family":"Yao","sequence":"first","affiliation":[{"name":"Faculty of Artificial Intelligence in Education, Central China Normal University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaomeng","family":"Yang","sequence":"additional","affiliation":[{"name":"Faculty of Artificial Intelligence in Education, Central China Normal University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Di","family":"Chen","sequence":"additional","affiliation":[{"name":"Faculty of Artificial Intelligence in Education, Central China Normal University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhao","family":"Wang","sequence":"additional","affiliation":[{"name":"Faculty of Artificial Intelligence in Education, Central China Normal University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuan","family":"Tian","sequence":"additional","affiliation":[{"name":"Faculty of Artificial Intelligence in Education, Central China Normal University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,7,30]]},"reference":[{"key":"ref_1","unstructured":"Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021, January 4\u20138). An image is worth 16 \u00d7 16 words: Transformers for image recognition at scale. Proceedings of the International Conference on Learning Representations (ICLR), Virtual."},{"key":"ref_2","unstructured":"Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and J\u00e9gou, H. (2021, January 18\u201324). Training data-efficient image transformers & distillation through attention. Proceedings of the International Conference on Machine Learning (ICML), Virtual."},{"key":"ref_3","unstructured":"Ma, F., Sun, B., and Li, S. (2021). Robust facial expression recognition with convolutional visual transformers. arXiv."},{"key":"ref_4","unstructured":"Li, H., Sui, M., Zhao, F., Zha, Z., and Wu, F. (2021). MVT: Mask vision transformer for facial expression recognition in the wild. arXiv."},{"key":"ref_5","first-page":"7","article-title":"Facial Expression recognition with visual transformers and attentional selective fusion","volume":"2","author":"Ma","year":"2021","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Dong, X., Bao, J., Chen, D., Zhang, W., Yu, N., Yuan, L., Chen, D., and Guo, B. (2022, January 21\u201324). CSWin Transformer: A general Vision Transformer backbone with cross-shaped windows. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.01181"},{"key":"ref_7","unstructured":"Krizhevsky, A., Sutskever, L., and Hinton, G. (2012, January 3\u20138). ImageNet classification with deep convolutional neural networks. Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS), Lake Tahoe, NV, USA."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Liu, S., and Deng, W. (2015, January 3\u20136). Very deep convolutional neural network based image classification using small training sample size. Proceedings of the Asian Conference on Pattern Recognition (ACPR), Lake Kuala Lumpur, Malaysia.","DOI":"10.1109\/ACPR.2015.7486599"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Li, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"ref_10","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), Las Vegas, NV, USA."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Pham, L., Vu, H., and Tran, T. (2021, January 18\u201321). Facial expression recognition using residual masking network. Proceedings of the International Conference on Pattern Recognition (ICPR), Taichung, Taiwan, China.","DOI":"10.1109\/ICPR48806.2021.9411919"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Mollahosseini, A., Chan, D., and Mahoor, M. (2016, January 7\u20139). Going deeper in facial expression recognition using deep neural networks. Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY, USA.","DOI":"10.1109\/WACV.2016.7477450"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Jung, H., Lee, S., Yim, J., Park, S., and Kim, J. (2015, January 13\u201316). Joint fine tuning in deep neural networks for facial expression recognition. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.341"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Yang, H., Ciftci, U., and Yin, L. (2018, January 18\u201322). Facial expression recognition by de-expression residue learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00231"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"610","DOI":"10.1016\/j.patcog.2016.07.026","article-title":"Facial expression recognition with convolutional neural networks: Coping with few data and the training sample order","volume":"61","author":"Lopes","year":"2017","journal-title":"Pattern Recognit."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"126","DOI":"10.1016\/j.neucom.2015.02.011","article-title":"AU-inspired deep networks for facial expression feature learning","volume":"159","author":"Liu","year":"2015","journal-title":"Neurocomputing"},{"key":"ref_17","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Jakob, U., Jones, L., Gomez, A.N., Kaiser, \u0141., and Polosukhin, I. (2017, January 4\u20139). Attention is all you need. Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS), Long Beach, CA, USA."},{"key":"ref_18","unstructured":"Aouayeb, M., Hamidouche, W., Soladie, C., Kpalma, K., and Seguier, R. (2021). Learning vision transformer with squeeze and excitation for facial expression recognition. arXiv."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"9995","DOI":"10.1109\/ACCESS.2023.3237817","article-title":"Fine-tuning SWin transformer and multiple weights optimality-seeking for facial expression recognition","volume":"11","author":"Feng","year":"2023","journal-title":"IEEE Access"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 11\u201317). SWin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Virtual.","DOI":"10.1109\/ICCV48922.2021.00986"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1109\/TAFFC.2017.2740923","article-title":"Affectnet: A database for facial expression, valence, and arousal computing in the wild","volume":"10","author":"Mollahosseini","year":"2017","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"2011","DOI":"10.1109\/TPAMI.2019.2913372","article-title":"Squeeze-and-excitation networks","volume":"42","author":"Hu","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Woo, S., Park, J., Lee, J.Y., and Kweon, I.S. (2018, January 8\u201314). CBAM: Convolutional block attention module. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01234-2_1"},{"key":"ref_24","unstructured":"Lian, D., Zhou, D., Feng, J., and Wang, X. (December, January 28). Scaling & shifting your features: A new baseline for efficient model tuning. Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), Virtual."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Li, S., Deng, W., and Du, J. (2017, January 21\u201326). Reliable crowd-sourcing and deep locality-preserving learning for expression recognition in the wild. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Hawaii, HO, USA.","DOI":"10.1109\/CVPR.2017.277"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Barsoum, E., Zhang, C., Ferrer, C.C., and Zhang, Z. (2016, January 12\u201316). Training deep networks for facial expression recognition with crowd-sourced label distribution. Proceedings of the ACM International Conference on Multimodal Interaction (ICMI), Tokyo, Japan.","DOI":"10.1145\/2993148.2993165"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"4057","DOI":"10.1109\/TIP.2019.2956143","article-title":"Region attention networks for pose and occlusion robust facial expression recognition","volume":"29","author":"Wang","year":"2020","journal-title":"IEEE Trans. Image Process."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"451","DOI":"10.1109\/TAFFC.2020.3031602","article-title":"Facial expression recognition in the wild using multi-level features and attention mechanisms","volume":"14","author":"Li","year":"2023","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Ding, H., Zhou, P., and Chellappa, R. (2020, January 27\u201330). Occlusion-adaptive deep network for robust facial expression recognition. Proceedings of the IEEE International Joint Conference on Biometrics (IJCB), Houston, TX, USA.","DOI":"10.1109\/IJCB48548.2020.9304923"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Wang, K., Peng, X., Yang, J., Lu, S., and Qiao, Y. (2020, January 14\u201319). Suppressing uncertainties for large-scale facial expression recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Virtual.","DOI":"10.1109\/CVPR42600.2020.00693"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Cai, J., Meng, Z., Khan, A.S., O\u2019Reilly, J., Li, Z., Han, S., and Tong, Y. (2021, January 19\u201322). Identity-free facial expression recognition using conditional generative adversarial network. Proceedings of the IEEE International Conference on Image Processing (ICIP), Anchorage, AK, USA.","DOI":"10.1109\/ICIP42928.2021.9506593"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Zhao, Z., Liu, Q., and Zhou, F. (2021, January 2\u20139). Robust lightweight facial expression recognition network with label distribution training. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), Vancouver, BC, Canada.","DOI":"10.1609\/aaai.v35i4.16465"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"781","DOI":"10.1016\/j.ins.2022.11.068","article-title":"Patch attention convolutional vision transformer for facial expression recognition with occlusion","volume":"619","author":"Liu","year":"2023","journal-title":"Inf. Sci."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1016\/j.ins.2021.08.043","article-title":"Facial expression recognition with grid-wise attention and visual transformer","volume":"580","author":"Huang","year":"2021","journal-title":"Inf. Sci."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Fan, X., Deng, Z., Wang, K., Peng, X., and Qiao, Y. (2020, January 25\u201328). Learning discriminative representation for facial expression recognition from uncertainties. Proceedings of the IEEE International Conference on Image Processing (ICIP), Abu Dhabi, United Arab Emirates.","DOI":"10.1109\/ICIP40778.2020.9190643"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Zeng, D., Lin, Z., Yan, X., Liu, Y., Wang, F., and Tang, B. (2022, January 21\u201324). Face2exp: Combating data biases for facial expression recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.01965"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"2439","DOI":"10.1109\/TIP.2018.2886767","article-title":"Occlusion aware facial expression recognition using CNN with attention mechanism","volume":"28","author":"Li","year":"2018","journal-title":"IEEE Trans. Image Process."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"033021","DOI":"10.1117\/1.JEI.30.3.033021","article-title":"TAMNET: Two attention modules-based network on facial expression recognition under uncertainty","volume":"30","author":"Shao","year":"2021","journal-title":"J. Electron. Imaging"},{"key":"ref_39","unstructured":"Han, Q., Fan, Z., Dai, Q., Sun, L., Cheng, M., Liu, J., and Wang, J. (2022, January 25\u201329). On the connection between local attention and dynamic depth-wise convolution. Proceedings of the International Conference on Learning Representations (ICLR), Virtual."},{"key":"ref_40","unstructured":"Zhou, J., Wang, P., Wang, F., Liu, Q., Li, H., and Rong, J. (2021). ELSA: Enhanced local self-attention for vision transformer. arXiv."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017, January 22\u201329). Grad-CAM: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.74"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"283","DOI":"10.1016\/j.patcog.2019.04.027","article-title":"Multi-resolution dictionary learning for face recognition","volume":"93","author":"Luo","year":"2019","journal-title":"Pattern Recognit."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/15\/6799\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:22:26Z","timestamp":1760127746000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/15\/6799"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,30]]},"references-count":42,"journal-issue":{"issue":"15","published-online":{"date-parts":[[2023,8]]}},"alternative-id":["s23156799"],"URL":"https:\/\/doi.org\/10.3390\/s23156799","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,7,30]]}}}