{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T04:34:31Z","timestamp":1760243671488,"version":"build-2065373602"},"reference-count":79,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2013,12,5]],"date-time":"2013-12-05T00:00:00Z","timestamp":1386201600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/3.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Over the last decade, human facial expressions recognition (FER) has emerged as an important research area. Several factors make FER a challenging research problem. These include varying light conditions in training and test images; need for automatic and accurate face detection before feature extraction; and high similarity among different expressions that makes it dif\ufb01cult to distinguish these expressions with a high accuracy. This work implements a hierarchical linear discriminant analysis-based facial expressions recognition (HL-FER) system to tackle these problems. Unlike the previous systems, the HL-FER uses a pre-processing step to eliminate light effects, incorporates a new automatic face detection scheme, employs methods to extract both global and local features, and utilizes a HL-FER to overcome the problem of high similarity among different expressions. Unlike most of the previous works that were evaluated using a single dataset, the performance of the HL-FER is assessed using three publicly available datasets under three different experimental settings: n-fold cross validation based on subjects for each dataset separately; n-fold cross validation rule based on datasets; and, \ufb01nally, a last set of experiments to assess the effectiveness of each module of the HL-FER separately. Weighted average recognition accuracy of 98.7% across three different datasets, using three classifiers, indicates the success of employing the HL-FER for human FER.<\/jats:p>","DOI":"10.3390\/s131216682","type":"journal-article","created":{"date-parts":[[2013,12,5]],"date-time":"2013-12-05T11:35:30Z","timestamp":1386243330000},"page":"16682-16713","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":32,"title":["Hierarchical Recognition Scheme for Human Facial Expression Recognition Systems"],"prefix":"10.3390","volume":"13","author":[{"given":"Muhammad","family":"Siddiqi","sequence":"first","affiliation":[{"name":"UC Lab, Department of Computer Engineering, Kyung Hee University, Yongin-Si 446-701, Korea"}]},{"given":"Sungyoung","family":"Lee","sequence":"additional","affiliation":[{"name":"UC Lab, Department of Computer Engineering, Kyung Hee University, Yongin-Si 446-701, Korea"}]},{"given":"Young-Koo","family":"Lee","sequence":"additional","affiliation":[{"name":"UC Lab, Department of Computer Engineering, Kyung Hee University, Yongin-Si 446-701, Korea"}]},{"given":"Adil","family":"Khan","sequence":"additional","affiliation":[{"name":"Division of Information and Computer Engineering, Ajou University, Suwon 443-749, Korea"}]},{"given":"Phan","family":"Truc","sequence":"additional","affiliation":[{"name":"UC Lab, Department of Computer Engineering, Kyung Hee University, Yongin-Si 446-701, Korea"}]}],"member":"1968","published-online":{"date-parts":[[2013,12,5]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1016\/S0031-3203(02)00052-3","article-title":"Automatic facial expression analysis: A survey","volume":"36","author":"Fasel","year":"2003","journal-title":"Patt. Recognit."},{"key":"ref_2","unstructured":"Bettadapura, V. (2012). Face Expression Recognition and Analysis: The State of the Art, Tech Report arXiv:1203.6722."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Wu, X., and Zhao, J. (2010, January 10\u201312). Curvelet Feature Extraction for Face Recognition and Facial Expression Recognition. Yantai, China.","DOI":"10.1109\/ICNC.2010.5583642"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"253","DOI":"10.1017\/S0048577299971664","article-title":"Measuring facial expressions by computer image analysis","volume":"36","author":"Bartlett","year":"1999","journal-title":"Psychophysiology"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Moore, S., Bowden, R., and Guildford, U. (2009, January 7\u203110). The Effects of Pose on Facial Expression Recognition. London, UK.","DOI":"10.5244\/C.23.79"},{"key":"ref_6","unstructured":"Gross, R., Baker, S., Matthews, I., and Kanade, T. (2005). Handbook of Face Recognition, Springer."},{"key":"ref_7","unstructured":"Zhu, Z., and Ji, Q. (2006, January 17\u201322). Robust Real-Time Face Pose and Facial Expression Recovery. New York, USA."},{"key":"ref_8","unstructured":"Bartlett, M., Littlewort, G., Frank, M., Lainscsek, C., Fasel, I., and Movellan, J. (2005, January 20\u201325). Recognizing Facial Expression: Machine Learning and Application to Spontaneous Behavior. San Diego, CA, USA."},{"key":"ref_9","first-page":"1271","article-title":"A Prototype for Automatic Recognition of Spontaneous Facial Actions","volume":"Volume 15","author":"Bartlett","year":"2003","journal-title":"Advances in Neural Information Processing Systems"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"22","DOI":"10.4304\/jmm.1.6.22-35","article-title":"Automatic recognition of facial actions in spontaneous expressions","volume":"1","author":"Bartlett","year":"2006","journal-title":"J. Multimed."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1186\/1475-925X-8-16","article-title":"Facial expression (mood) recognition from facial images using committee neural networks","volume":"10","author":"Kulkarni","year":"2009","journal-title":"BioMed. Eng. OnLine"},{"key":"ref_12","first-page":"389","article-title":"Automatic recognition and analysis of human faces and facial expression by LDA using wavelet transform","volume":"67","author":"Sumathi","year":"2012","journal-title":"Eur. J. Sci. Res."},{"key":"ref_13","first-page":"546","article-title":"Facial expression recognition with local binary patterns and linear programming","volume":"15","author":"Feng","year":"2005","journal-title":"Pattern Recognition And Image Analysis C\/C of Raspoznavaniye Obrazov I Analiz Izobrazhenii"},{"key":"ref_14","first-page":"24","article-title":"Recognition of facial expressions with principal component analysis and singular value decomposition","volume":"9","author":"Kaur","year":"2010","journal-title":"Int. J. Comput. Appl."},{"key":"ref_15","unstructured":"Gupta, S., Agrwal, S., Meena, Y., and Nain, N. (December, January 28). A Hybrid Method of Feature Extraction for Facial Expression Recognition. Dijon, France."},{"key":"ref_16","unstructured":"Gonzalez, R.C., and Woods, R.E. (2008). Digital Image Processing, Prentice Hall."},{"key":"ref_17","unstructured":"Eramian, M., and Mould, D. (2005, January 9\u201311). Histogram Equalization Using Neighborhood Metrics. British Columbia, Canada."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"891","DOI":"10.1109\/TCE.2009.5174471","article-title":"Image sharpening using sub-regions histogram equalization","volume":"55","author":"Ibrahim","year":"2009","journal-title":"IEEE Trans. Consum. Electron."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Yeganeh, H., Ziaei, A., and Rezaie, A. (2008, January 13\u201315). A Novel Approach for Contrast Enhancement Based on Histogram Equalization. Kuala Lumpur, Malaysia.","DOI":"10.1109\/ICCCE.2008.4580607"},{"key":"ref_20","first-page":"61","article-title":"Image inversion and Bi level histogram equalization for contrast enhancement","volume":"1","author":"Shanmugavadivu","year":"2010","journal-title":"Int. J. Comput. Appl."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"425","DOI":"10.1016\/j.ijleo.2011.12.057","article-title":"Range limited Bi-histogram equalization for image contrast enhancement","volume":"124","author":"Zuo","year":"2013","journal-title":"Optik"},{"key":"ref_22","unstructured":"Beham, M.P., and Roomi, S.M.M. (2012, January 24\u201325). Face recognition using appearance based approach: A literature survey. Mumbai, Maharashtra, India."},{"key":"ref_23","first-page":"181","article-title":"A hybrid face detection system using combination of appearance-based and feature-based methods","volume":"9","author":"Tabatabaie","year":"2009","journal-title":"Int. J. Comput. Sci. Netw. Secur."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1740","DOI":"10.1109\/TIP.2012.2235848","article-title":"Local directional number pattern for face analysis: Face and expression recognition","volume":"22","author":"Rivera","year":"2013","journal-title":"IEEE Trans. Image Process."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1109\/34.655647","article-title":"Neural network-based face detection","volume":"20","author":"Rowley","year":"1998","journal-title":"IEEE Trans. PAMI."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"875","DOI":"10.1109\/TC.1973.5009188","article-title":"Angle detection on digital curves","volume":"22","author":"Rosenfeld","year":"1973","journal-title":"IEEE Trans. Comput."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1033","DOI":"10.1016\/S0031-3203(00)00042-X","article-title":"Multi-cues eye detection on gray intensity image","volume":"34","author":"Feng","year":"2001","journal-title":"Pattern Recognit."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1056","DOI":"10.1016\/j.engappai.2007.11.010","article-title":"Recognition of facial expressions using Gabor wavelets and learning vector quantization","volume":"21","author":"Bashyal","year":"2008","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Jabid, T., Kabir, M., and Chae, O. (2010, January 26\u201329). Facial Expression Recognition Using Local Directional Pattern (LDP). Hong Kong.","DOI":"10.1109\/ICIP.2010.5652374"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Rahman, A., and Ali, L. (2011, January 12\u201314). Weighted Local Directional Pattern for Robust Facial Expression Recognition. Bandung, Indonesia.","DOI":"10.1109\/ICI.2011.51"},{"key":"ref_31","first-page":"382","article-title":"Local Directional Pattern Variance (LDPv): A robust feature descriptor for facial expression recognition","volume":"9","author":"Kabir","year":"2012","journal-title":"Int. Arab J. Inf. Technol."},{"key":"ref_32","unstructured":"Gourier, N., Hall, D., and Crowley, J. (2004). FG Net Workshop on Visual Observation of Deictic Gestures, FGnet (IST\u20132000\u201326434)."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Cox, I., Ghosn, J., and Yianilos, P. (1996, January 18\u201320). Feature-Based Face Recognition Using Mixture-Distance. 1996. San Francisco, CA, USA.","DOI":"10.1109\/CVPR.1996.517076"},{"key":"ref_34","first-page":"1033","article-title":"Facial expression classification using PCA and hierarchical radial basis function network","volume":"22","author":"Lin","year":"2006","journal-title":"J. Inf. Sci. Eng."},{"key":"ref_35","unstructured":"Lin, D.T. (2006). Artificial Neural Networks\u2013ICANN 2006, Springer."},{"key":"ref_36","first-page":"341","article-title":"Facial expression recognition by supervised independent component analysis using MAP estimation","volume":"91","author":"Fan","year":"2008","journal-title":"IEICE Trans. Inf. Syst."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Buciu, I., and Pitas, I. (2004, January 23\u201326). Application of Non-Negative and Local Non Negative Matrix Factorization to Facial Expression Recognition. Cambridge, UK.","DOI":"10.1109\/ICPR.2004.1334109"},{"key":"ref_38","unstructured":"Shinohara, Y., and Otsuf, N. (2004, January 17\u201319). Facial Expression Recognition Using fisher Weight Maps. Seoul, South Korea."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1357","DOI":"10.1109\/34.817413","article-title":"Automatic classification of single facial images","volume":"21","author":"Lyons","year":"1999","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_40","unstructured":"He, L., Zhou, J., Hu, D., Zou, C., and Zhao, L. (June, January 30). Boosted Independent Features for Face Expression Recognition. Chongqing, China."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Lirong, W., Xiaoguang, Y., Jianlei, W., and Jian, Z. (2011, January 24\u201326). Facial Expression Recognition Based on Local Texture Features. Dalian, China.","DOI":"10.1109\/CSE.2011.96"},{"key":"ref_42","first-page":"113","article-title":"Automatic facial feature extraction and expression recognition based on neural network","volume":"2","author":"Khandait","year":"2012","journal-title":"Int. J. Adv. Comput. Sci. Appl."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"903","DOI":"10.1109\/TIP.2008.2011167","article-title":"Iterative subspace analysis based on feature line distance","volume":"18","author":"Pang","year":"2009","journal-title":"IEEE Trans. Image Process."},{"key":"ref_44","first-page":"20","article-title":"Eigenface-gabor algorithm for feature extraction in face recognition","volume":"3","author":"Perez","year":"2009","journal-title":"Int. J. Comput."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"1674","DOI":"10.3844\/jcssp.2012.1674.1679","article-title":"Facial expressions recognition using eigenspaces","volume":"8","author":"Kittusamy","year":"2012","journal-title":"J. Comput. Sci."},{"key":"ref_46","first-page":"196","article-title":"Recognition of facial expression using eigenvector based distributed features and euclidean distance based decision making technique","volume":"4","author":"Kalita","year":"2013","journal-title":"Int. J. Adv. Comput. Sci. Appl."},{"key":"ref_47","first-page":"30","article-title":"A neural network based facial expression recognition using sherface","volume":"59","author":"Abidin","year":"2012","journal-title":"Int. J. Comput. Appl."},{"key":"ref_48","first-page":"653","article-title":"A literature survey on facial expression recognition using global features","volume":"2","author":"Mistry","year":"2013","journal-title":"Int. J. Eng. Adv. Technol."},{"key":"ref_49","unstructured":"Tian, Y., Kanade, T., and Cohn, J. (2005). Handbook of Face Recognition, Springer. Chapter 11."},{"key":"ref_50","first-page":"399","article-title":"Conditional Mutual Information Based Boosting for Facial Expression Recognition","volume":"Volume 1","author":"Shan","year":"2005","journal-title":"British Machine Vision Conference (BMVC)"},{"key":"ref_51","first-page":"263","article-title":"Emotion recognition system by a neural network based facial expression analysis","volume":"54","author":"Filko","year":"2013","journal-title":"Autom.\u2013J. Control Meas. Electron. Comput. Commun."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"1225","DOI":"10.1016\/S0895-4356(96)00002-9","article-title":"Advantages and disadvantages of using artificial neural networks versus logistic regression for predicting medical outcomes","volume":"49","author":"Tu","year":"1996","journal-title":"J. Clin. Epidemiol."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1109\/TIP.2006.884954","article-title":"Facial expression recognition in image sequences using geometric deformation features and support vector machines","volume":"16","author":"Kotsia","year":"2007","journal-title":"IEEE Trans. Image Process."},{"key":"ref_54","first-page":"650","article-title":"Human emotion recognition system using optimally designed SVM with different facial feature extraction techniques","volume":"7","author":"Kharat","year":"2008","journal-title":"WSEAS Trans. Comput."},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Tsai, H., Lai, Y., and Zhang, Y. (2010, January 11\u201314). Using SVM to Design Facial Expression Recognition for Shape and Texture Features. Qingdao, China.","DOI":"10.1109\/ICMLC.2010.5580938"},{"key":"ref_56","unstructured":"Chen, F., and Kotani, K. (2007, January 16\u201318). Facial Expression Recognition by SVM-based Two-Stage Classifier on Gabor Features. Tokyo, Japan."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"803","DOI":"10.1016\/j.imavis.2008.08.005","article-title":"Facial expression recognition based on local binary patterns: A comprehensive study","volume":"27","author":"Shan","year":"2009","journal-title":"Image Vis. Comput."},{"key":"ref_58","unstructured":"Support Vector Machines. Available online: http:\/\/scikit-learn.org\/stable\/modules\/svm.html."},{"key":"ref_59","doi-asserted-by":"crossref","unstructured":"Liu, W., Lu, J., Wang, Z., and Song, H. (2008, January 27\u201330). An Expression Space Model for Facial Expression Analysis. Sanya, China.","DOI":"10.1109\/CISP.2008.216"},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"Schels, M., and Schwenker, F. (2010, January 23\u201326). A Multiple Classifier System Approach for Facial Expressions in Image Sequences Utilizing GMM Supervectors. Istanbul, Turkey.","DOI":"10.1109\/ICPR.2010.1033"},{"key":"ref_61","first-page":"265","article-title":"Modeling of dynamic backgrounds by type-2 fuzzy Gaussians mixture models","volume":"1","author":"Bouwmans","year":"2009","journal-title":"MASAUM J. Basic Appl. Sci."},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"500","DOI":"10.1109\/TMM.2006.870737","article-title":"Recognition of facial expressions and measurement of levels of interest from video","volume":"8","author":"Yeasin","year":"2006","journal-title":"IEEE Trans. Multimed."},{"key":"ref_63","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1109\/TIFS.2005.863510","article-title":"Automatic facial expression recognition using facial animation parameters and multistream HMMs","volume":"1","author":"Aleksic","year":"2006","journal-title":"IEEE Trans. Inf. Forensics Secur."},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"2216","DOI":"10.1109\/TCE.2009.5373791","article-title":"An enhanced independent component-based human facial expression recognition from video","volume":"55","author":"Uddin","year":"2009","journal-title":"IEEE Trans. Consum. Electron."},{"key":"ref_65","doi-asserted-by":"crossref","unstructured":"Mariappan, M.B., Suk, M., and Prabhakaran, B. (2012, January 10\u201312). Facial Expression Recognition using Dual Layer Hierarchical SVM Ensemble Classification. Irvine, CA, USA.","DOI":"10.1109\/ISM.2012.104"},{"key":"ref_66","doi-asserted-by":"crossref","first-page":"2757","DOI":"10.1016\/S0031-3203(03)00175-4","article-title":"Constructing support vector machine ensemble","volume":"36","author":"Kim","year":"2003","journal-title":"Pattern Recognit."},{"key":"ref_67","first-page":"227","article-title":"A robust skin color based face detection algorithm","volume":"6","author":"Singh","year":"2003","journal-title":"Tamkang J. Sci. Eng."},{"key":"ref_68","doi-asserted-by":"crossref","first-page":"1317","DOI":"10.1016\/S0031-3203(98)00162-9","article-title":"Locating head and face boundaries for head shoulder images","volume":"32","author":"Hu","year":"1999","journal-title":"Pattern Recognit."},{"key":"ref_69","unstructured":"Starovoitov, V.V., Samal, D.I., and Sankur, B. (2000, January 5\u20139). Matching of Faces in Camera Images and Document Photographs. Istanbul, Turkey."},{"key":"ref_70","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1167\/8.8.1","article-title":"The contribution of different facial regions to the recognition of conversational expressions","volume":"8","author":"Nusseck","year":"2008","journal-title":"J. Vis."},{"key":"ref_71","doi-asserted-by":"crossref","first-page":"e32321","DOI":"10.1371\/journal.pone.0032321","article-title":"The MPI facial expression database a validated database of emotional and conversational facial expressions","volume":"7","author":"Kaulard","year":"2012","journal-title":"PLoS One"},{"key":"ref_72","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1002\/ajpa.20001","article-title":"Human facial expressions as adaptations: Evolutionary questions in facial expression research","volume":"116","author":"Schmidt","year":"2002","journal-title":"Am. J. Phys. Anthropol."},{"key":"ref_73","unstructured":"Kanade, T., Cohn, J.F., and Tian, Y. (2000, January 28\u201330). Comprehensive Database for Facial Expression Analysis. Grenoble, France."},{"key":"ref_74","unstructured":"Lyons, M., Akamatsu, S., Kamachi, M., and Gyoba, J. (1998, January 14\u201316). Coding Facial Expressions with Gabor Wavelets. Nara, Japan."},{"key":"ref_75","unstructured":"Samaria, F.S., and Harter, A. (1994, January 5\u20137). Parameterization of a Stochastic Model for Human Face Identification. Sarasota, FL, USA."},{"key":"ref_76","doi-asserted-by":"crossref","first-page":"643","DOI":"10.1109\/34.927464","article-title":"From few to many: Illumination cone models for face recognition under variable lighting and pose","volume":"23","author":"Georghiades","year":"2001","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_77","doi-asserted-by":"crossref","first-page":"225","DOI":"10.1007\/s11263-008-0143-7","article-title":"Evaluation of face datasets as tools for assessing the performance of face recognition methods","volume":"79","author":"Shamir","year":"2008","journal-title":"Int. J. Comput. Vis."},{"key":"ref_78","unstructured":"Zhou, J., Xu, T., and Gan, J. (2013, January 8\u201310). Facial Expression Recognition Based on Local Directional Pattern Using SVM Decision-Level Fusion. Nancy, France."},{"key":"ref_79","doi-asserted-by":"crossref","first-page":"9573","DOI":"10.3390\/s111009573","article-title":"Facial expression recognition based on local binary patterns and kernel discriminant isomap","volume":"11","author":"Zhao","year":"2011","journal-title":"Sensors"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/13\/12\/16682\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T21:51:08Z","timestamp":1760219468000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/13\/12\/16682"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,12,5]]},"references-count":79,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2013,12]]}},"alternative-id":["s131216682"],"URL":"https:\/\/doi.org\/10.3390\/s131216682","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2013,12,5]]}}}