{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,1]],"date-time":"2025-12-01T15:39:03Z","timestamp":1764603543506,"version":"build-2065373602"},"reference-count":51,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2019,1,5]],"date-time":"2019-01-05T00:00:00Z","timestamp":1546646400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Zhejiang Provincial National Science Foundation of China","award":["LY16F020011","LY14F020036"],"award-info":[{"award-number":["LY16F020011","LY14F020036"]}]},{"name":"National Science Foundation of China (NSFC)","award":["61203257"],"award-info":[{"award-number":["61203257"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Symmetry"],"abstract":"<jats:p>The proposed method has 30 streams, i.e., 15 spatial streams and 15 temporal streams. Each spatial stream corresponds to each temporal stream. Therefore, this work correlates with the symmetry concept. It is a difficult task to classify video-based facial expression owing to the gap between the visual descriptors and the emotions. In order to bridge the gap, a new video descriptor for facial expression recognition is presented to aggregate spatial and temporal convolutional features across the entire extent of a video. The designed framework integrates a state-of-the-art 30 stream and has a trainable spatial\u2013temporal feature aggregation layer. This framework is end-to-end trainable for video-based facial expression recognition. Thus, this framework can effectively avoid overfitting to the limited emotional video datasets, and the trainable strategy can learn to better represent an entire video. The different schemas for pooling spatial\u2013temporal features are investigated, and the spatial and temporal streams are best aggregated by utilizing the proposed method. The extensive experiments on two public databases, BAUM-1s and eNTERFACE05, show that this framework has promising performance and outperforms the state-of-the-art strategies.<\/jats:p>","DOI":"10.3390\/sym11010052","type":"journal-article","created":{"date-parts":[[2019,1,9]],"date-time":"2019-01-09T03:06:06Z","timestamp":1547003166000},"page":"52","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":18,"title":["Deep Temporal\u2013Spatial Aggregation for Video-Based Facial Expression Recognition"],"prefix":"10.3390","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0469-7178","authenticated-orcid":false,"given":"Xianzhang","family":"Pan","sequence":"first","affiliation":[{"name":"Institute of Intelligent Information Processing, Taizhou University, Taizhou 318000, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0405-1775","authenticated-orcid":false,"given":"Wenping","family":"Guo","sequence":"additional","affiliation":[{"name":"Institute of Intelligent Information Processing, Taizhou University, Taizhou 318000, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiaoying","family":"Guo","sequence":"additional","affiliation":[{"name":"School of Software Engineering, Institute of Big Data Science and Industry, Taiyuan University, Shanxi 030006, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wenshu","family":"Li","sequence":"additional","affiliation":[{"name":"College of information science and technology, Zhejiang Sci-Tech University, Hangzhou 310018, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Junjie","family":"Xu","sequence":"additional","affiliation":[{"name":"College of Computer Science and Technology, Civil Aviation University of China, Tianjin 300300, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jinzhao","family":"Wu","sequence":"additional","affiliation":[{"name":"Guangxi Key Laboratory of Hybrid Computation and IC Design Analysis, Guangxi University for Nationalities, Nanning 530006, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2019,1,5]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"3030","DOI":"10.1109\/TCSVT.2017.2719043","article-title":"Learning Affective Features with a Hybrid Deep Model for Audio-Visual Emotion Recognition","volume":"28","author":"Zhang","year":"2018","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_2","first-page":"455","article-title":"Facial expression of pain: An evolutionary account","volume":"25","author":"Williams","year":"2002","journal-title":"Behav. Brain Sci."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Zhao, H., Sun, M., Deng, W., and Yang, X. (2017). A New Feature Extraction Method Based on EEMD and Multi-Scale Fuzzy Entropy for Motor Bearing. Entropy, 19.","DOI":"10.3390\/e19010014"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1109\/MPRV.2010.46","article-title":"Facial expression analysis for predicting unsafe driving behavior","volume":"10","author":"Jabon","year":"2011","journal-title":"IEEE Pervasive Comput."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"35042","DOI":"10.1109\/ACCESS.2018.2834540","article-title":"A novel fault diagnosis method based on integrating empirical wavelet transform and fuzzy entropy for motor bearing","volume":"6","author":"Deng","year":"2018","journal-title":"IEEE Access"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Leo, M., Carcagn\u00ec, P., Distante, C., Spagnolo, P., Mazzeo, P., Rosato, A., Petrocchi, S., Pellegrino, C., Levante, A., and De Lum\u00e8, F. (2018). Computational Assessment of Facial Expression Production in ASD Children. Sensors, 18.","DOI":"10.3390\/s18113993"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"384","DOI":"10.1037\/0003-066X.48.4.384","article-title":"Facial expression and emotion","volume":"48","author":"Ekman","year":"1993","journal-title":"Am. Psychol."},{"key":"ref_8","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3\u20136). ImageNet classification with deep convolutional neural networks. Proceedings of the International Conference on Neural Information Processing Systems, Lake Tahoe, NV, USA."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"300","DOI":"10.1109\/TAFFC.2016.2553038","article-title":"BAUM-1: A Spontaneous Audio-Visual Face Database of Affective and Mental States","volume":"8","author":"Zhalehpour","year":"2017","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Martin, O., Kotsia, I., Macq, B., and Pitas, I. (2006, January 3\u20137). The eNTERFACE\u2019 05 Audio-Visual Emotion Database. Proceedings of the International Conference on Data Engineering Workshops, Atlanta, GA, USA.","DOI":"10.1109\/ICDEW.2006.145"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Ren, Z., Skjetne, R., and Gao, Z. (2019). A Crane Overload Protection Controller for Blade Lifting Operation Based on Model Predictive Control. Energies, 12.","DOI":"10.3390\/en12010050"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"2816","DOI":"10.1109\/TMM.2017.2713408","article-title":"Multimodal 2D+3D Facial Expression Recognition with Deep Fusion Convolutional Neural Network","volume":"19","author":"Huibin","year":"2017","journal-title":"IEEE Trans. Multimed."},{"key":"ref_13","unstructured":"Liu, M., Li, S., Shan, S., Wang, R., and Chen, X. (2014, January 1\u20135). Deeply Learning Deformable Facial Action Parts Model for Dynamic Expression Analysis. Proceedings of the Asian Conference on Computer Vision, Singapore."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"4193","DOI":"10.1109\/TIP.2017.2689999","article-title":"Facial Expression Recognition Based on Deep Evolutional Spatial-Temporal Networks","volume":"26","author":"Zhang","year":"2017","journal-title":"IEEE Trans. Image Process."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1839","DOI":"10.1109\/TCSVT.2017.2682196","article-title":"Pooling the Convolutional Layers in Deep ConvNets for Video Action Recognition","volume":"28","author":"Zhao","year":"2018","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Zhao, H., Yao, R., Xu, L., Yuan, Y., Li, G., and Deng, W. (2018). Study on a Novel Fault Damage Degree Identification Method Using High-Order Differential Mathematical Morphology Gradient Spectrum Entropy. Entropy, 20.","DOI":"10.3390\/e20090682"},{"key":"ref_17","unstructured":"Arandjelovic, R., Gronat, P., Torii, A., Pajdla, T., and Sivic, J. (July, January 26). NetVLAD: CNN architecture for weakly supervised place recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Borza, D., Danescu, R., Itu, R., and Darabant, A. (2017). High-Speed Video System for Micro-Expression Detection and Recognition. Sensors, 17.","DOI":"10.3390\/s17122913"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Liu, Y., Li, Y., Ma, X., and Song, R. (2017). Facial Expression Recognition with Fusion Features Extracted from Salient Facial Areas. Sensors, 17.","DOI":"10.20944\/preprints201701.0102.v1"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Xie, W., Shen, L., Yang, M., and Lai, Z. (2017). Active AU Based Patch Weighting for Facial Expression Recognition. Sensors, 17.","DOI":"10.20944\/preprints201701.0120.v1"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Sikka, K., Wu, T., Susskind, J., and Bartlett, M. (2012, January 16\u201318). Exploring bag of words architectures in the facial expression domain. Proceedings of the International Conference on Computer Vision, Xiamen, China.","DOI":"10.1007\/978-3-642-33868-7_25"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"756","DOI":"10.1016\/j.neucom.2015.09.083","article-title":"Facial expression recognition using sparse local Fisher discriminant analysis","volume":"174","author":"Wang","year":"2016","journal-title":"Neurocomputing"},{"key":"ref_23","unstructured":"Lyons, M., Akamatsu, S., Kamachi, M., and Gyoba, J. (1998, January 14\u201316). Coding facial expressions with Gabor wavelets. Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition, Nara, Japan."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"2254","DOI":"10.1016\/j.patrec.2012.07.007","article-title":"Local phase quantization descriptor for improving shape retrieval\/classification","volume":"33","author":"Nanni","year":"2012","journal-title":"Pattern Recognit. Lett."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Kayaoglu, M., and Erdem, C.E. (2015, January 9\u201313). Affect Recognition using Key Frame Selection based on Minimum Sparse Reconstruction. Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, Seattle, WA, USA.","DOI":"10.1145\/2818346.2830594"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"504","DOI":"10.1126\/science.1127647","article-title":"Reducing the dimensionality of data with neural networks","volume":"313","author":"Hinton","year":"2006","journal-title":"Science"},{"key":"ref_27","first-page":"59","article-title":"Development of Speech Control for Robotic Hand Using Neural Network and Stream Processing Method","volume":"9","author":"Haryanto","year":"2017","journal-title":"Internetworking Indones. J."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"565","DOI":"10.1016\/j.asoc.2018.01.008","article-title":"Adaptive Neuro-Fuzzy Inference System for Deburring Stage Classification and Prediction for Indirect Quality Monitoring","volume":"72","author":"Caesarendra","year":"2018","journal-title":"Appl. Soft Comput."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"557","DOI":"10.1016\/j.triboint.2017.06.032","article-title":"The determination of combustion engine condition and reliability using oil analysis by MLP and RBF neural networks","volume":"115","author":"Gajewski","year":"2017","journal-title":"Tribol. Int."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"3127","DOI":"10.1007\/s00170-017-1430-7","article-title":"Data mining tools in identifying the components of the microstructure of compacted graphite iron based on the content of alloying elements","volume":"95","author":"Regulski","year":"2018","journal-title":"Int. J. Adv. Manuf. Technol."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Kim, B.K., Lee, H., Roh, J., and Lee, S.Y. (2015, January 9\u201313). Hierarchical Committee of Deep CNNs with Exponentially-Weighted Decision Fusion for Static Facial Expression Recognition. Proceedings of the ACM on International Conference on Multimodal Interaction, Seattle, WA, USA.","DOI":"10.1145\/2818346.2830590"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Deng, W., Yao, R., Zhao, H.M., Yang, X.H., and Li, G.Y. (2017). A novel intelligent diagnosis method using optimal LS-SVM with improved PSO algorithm. Soft Comput., 1\u201318.","DOI":"10.1007\/s00500-017-2940-9"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Mollahosseini, A., Chan, D., and Mahoor, M.H. (2016, January 7\u201310). Going Deeper in Facial Expression Recognition using Deep Neural Networks. Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY, USA.","DOI":"10.1109\/WACV.2016.7477450"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Ding, H., Zhou, S.K., Chellappa, R., Ding, H., Zhou, S.K., and Chellappa, R. (June, January 30). FaceNet2ExpNet: Regularizing a Deep Face Recognition Net for Expression Recognition. Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), Washington, DC, USA.","DOI":"10.1109\/FG.2017.23"},{"key":"ref_35","unstructured":"Kahou, S.E., Michalski, V., Konda, K., Memisevic, R., and Pal, C. (2015, January 9\u201313). Recurrent Neural Networks for Emotion Recognition in Video. Proceedings of the ACM International Conference on Multimodal Interaction, Seattle, WA, USA."},{"key":"ref_36","unstructured":"Rodriguez, P., Cucurull, G., Gonzalez, J., Gonfaus, J.M., Nasrollahi, K., Moeslund, T.B., and Roca, F.X. (2017). Deep Pain: Exploiting Long Short-Term Memory Networks for Facial Expression Classification. IEEE Trans. Syst. Man Cybern."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Gao, J., Fu, Y., Jiang, Y.G., and Xue, X. (2017, January 6\u20139). Frame-Transformer Emotion Classification Network. Proceedings of the ACM on International Conference on Multimedia Retrieval, New York, NY, USA.","DOI":"10.1145\/3078971.3079030"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"42532","DOI":"10.1109\/ACCESS.2018.2858278","article-title":"Geometric-Convolutional Feature Fusion Based on Learning Propagation for Facial Expression Recognition","volume":"6","author":"Tang","year":"2018","journal-title":"IEEE Access"},{"key":"ref_39","unstructured":"Kim, D.H., Baddar, W., Jang, J., and Yong, M.R. (2017). Multi-Objective based Spatio-Temporal Feature Representation Learning Robust to Expression Intensity Variations for Facial Expression Recognition. IEEE Trans. Affect. Comput."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Ballester, P.L., and Araujo, R.M. (2016, January 12\u201317). On the performance of GoogLeNet and AlexNet applied to sketches. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.","DOI":"10.1609\/aaai.v30i1.10171"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1023\/B:VISI.0000045324.43199.43","article-title":"Lucas\/Kanade Meets Horn\/Schunck: Combining Local and Global Optic Flow Methods","volume":"61","author":"Bruhn","year":"2005","journal-title":"Int. J. Comput. Vis."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1023\/B:VISI.0000013087.49260.fb","article-title":"Robust Real-time Face Detection","volume":"57","author":"Viola","year":"2004","journal-title":"Int. J. Comput. Vis."},{"key":"ref_43","first-page":"1","article-title":"Learning Discriminative Dictionary for Facial Expression Recognition","volume":"33","author":"Zhang","year":"2017","journal-title":"IETE Tech. Rev."},{"key":"ref_44","unstructured":"M\u00fcller, C. (2010, January 26\u201330). The INTERSPEECH 2010 Paralinguistic Challenge. Proceedings of the Interspeech, Chiba, Japan."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"288","DOI":"10.1016\/j.asoc.2017.06.004","article-title":"Study on an improved adaptive PSO algorithm for solving multi-objective gate assignment","volume":"59","author":"Deng","year":"2017","journal-title":"Appl. Soft Comput."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s00500-016-2071-8","article-title":"A novel collaborative optimization algorithm in solving complex optimization problems","volume":"21","author":"Deng","year":"2017","journal-title":"Soft Comput."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"957","DOI":"10.1109\/TPAMI.2005.127","article-title":"Sparse multinomial logistic regression: Fast algorithms and generalization bounds","volume":"27","author":"Krishnapuram","year":"2005","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Klaser, A., Marsza\u0142ek, M., and Schmid, C. (2008, January 1\u20134). A Spatio-Temporal Descriptor based on 3D Gradients (HOG3D). Proceedings of the BMVC 2008\u201419th British Machine Vision Conference, Leeds, UK.","DOI":"10.5244\/C.22.99"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1109\/TPAMI.2012.59","article-title":"3D Convolutional Neural Networks for Human Action Recognition","volume":"35","author":"Ji","year":"2013","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"277","DOI":"10.1007\/s11042-009-0344-2","article-title":"Multimodal information fusion application to human emotion recognition from face and speech","volume":"49","author":"Mansoorizadeh","year":"2010","journal-title":"Multimed. Tools Appl."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"399","DOI":"10.1007\/s00521-012-1228-3","article-title":"Audiovisual emotion recognition using ANOVA feature selection method and multi-classifier neural networks","volume":"24","author":"Bejani","year":"2014","journal-title":"Neural Comput. Appl."}],"container-title":["Symmetry"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-8994\/11\/1\/52\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T12:23:45Z","timestamp":1760185425000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-8994\/11\/1\/52"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,1,5]]},"references-count":51,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2019,1]]}},"alternative-id":["sym11010052"],"URL":"https:\/\/doi.org\/10.3390\/sym11010052","relation":{},"ISSN":["2073-8994"],"issn-type":[{"type":"electronic","value":"2073-8994"}],"subject":[],"published":{"date-parts":[[2019,1,5]]}}}