{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T23:54:18Z","timestamp":1772236458148,"version":"3.50.1"},"reference-count":68,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2022,5,31]],"date-time":"2022-05-31T00:00:00Z","timestamp":1653955200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Automatic identification of human facial expressions has many potential applications in today\u2019s connected world, from mental health monitoring to feedback for onscreen content or shop windows and sign-language prosodic identification. In this work we use visual information as input, namely, a dataset of face points delivered by a Kinect device. The most recent work on facial expression recognition uses Machine Learning techniques, to use a modular data-driven path of development instead of using human-invented ad hoc rules. In this paper, we present a Machine-Learning based method for automatic facial expression recognition that leverages information fusion architecture techniques from our previous work and soft voting. Our approach shows an average prediction performance clearly above the best state-of-the-art results for the dataset considered. These results provide further evidence of the usefulness of information fusion architectures rather than adopting the default ML approach of features aggregation.<\/jats:p>","DOI":"10.3390\/s22114206","type":"journal-article","created":{"date-parts":[[2022,6,1]],"date-time":"2022-06-01T21:43:42Z","timestamp":1654119822000},"page":"4206","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Facial Expression Recognition from Multi-Perspective Visual Inputs and Soft Voting"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5155-3543","authenticated-orcid":false,"given":"Antonio A.","family":"Aguileta","sequence":"first","affiliation":[{"name":"Facultad de Matem\u00e1ticas, Universidad Aut\u00f3noma de Yucat\u00e1n, M\u00e9rida 97110, Mexico"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0995-2273","authenticated-orcid":false,"given":"Ram\u00f3n F.","family":"Brena","sequence":"additional","affiliation":[{"name":"School of Engineering and Sciences, Tecnologico de Monterrey, Monterrey 64849, Mexico"},{"name":"Departamento de Computaci\u00f3n y Dise\u00f1o, Instituto Tecnol\u00f3gico de Sonora, Ciudad Obreg\u00f3n 85000, Mexico"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7615-4431","authenticated-orcid":false,"given":"Erik","family":"Molino-Minero-Re","sequence":"additional","affiliation":[{"name":"Instituto de Investigaciones en Matem\u00e1ticas Aplicadas y en Sistemas\u2014Unidad Yucat\u00e1n, Universidad Nacional Aut\u00f3noma de M\u00e9xico, Sierra Papacal, Yucat\u00e1n 97302, Mexico"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7635-4687","authenticated-orcid":false,"given":"Carlos E.","family":"Galv\u00e1n-Tejada","sequence":"additional","affiliation":[{"name":"Unidad Acad\u00e9mica de Ingenier\u00eda El\u00e9ctrica y Comunicaciones, Universidad Aut\u00f3noma de Zacatecas, Zacatecas 98000, Mexico"}]}],"member":"1968","published-online":{"date-parts":[[2022,5,31]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"399","DOI":"10.1145\/954339.954342","article-title":"Face recognition: A literature survey","volume":"35","author":"Zhao","year":"2003","journal-title":"ACM Comput. Surv."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"509","DOI":"10.1007\/s00371-020-02031-z","article-title":"An automated and efficient convolutional architecture for disguise-invariant face recognition using noise-based data augmentation and deep transfer learning","volume":"38","author":"Khan","year":"2022","journal-title":"Vis. Comput."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1109\/MSECP.2003.1193209","article-title":"Biometric recognition: Security and privacy concerns","volume":"1","author":"Prabhakar","year":"2003","journal-title":"IEEE Secur. Priv."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Ko, B.C. (2018). A brief review of facial emotion recognition based on visual information. Sensors, 18.","DOI":"10.3390\/s18020401"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"445","DOI":"10.1007\/s00371-018-1607-6","article-title":"Micro-expression recognition: An updated review of current trends, challenges and solutions","volume":"36","author":"Goh","year":"2020","journal-title":"Vis. Comput."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1177\/1529100619832930","article-title":"Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements","volume":"20","author":"Barrett","year":"2019","journal-title":"Psychol. Sci. Public Interest"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"315","DOI":"10.1016\/j.ipm.2008.09.003","article-title":"Acoustic feature selection for automatic emotion recognition from speech","volume":"45","author":"Rong","year":"2009","journal-title":"Inf. Process. Manag."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Glowinski, D., Camurri, A., Volpe, G., Dael, N., and Scherer, K. (2008, January 23\u201328). Technique for automatic emotion recognition by body gesture analysis. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Anchorage, AK, USA.","DOI":"10.1109\/CVPRW.2008.4563173"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1243","DOI":"10.1007\/s10796-017-9765-z","article-title":"Grammatical facial expression recognition in sign language discourse: A study at the syntax level","volume":"19","author":"Freitas","year":"2017","journal-title":"Inf. Syst. Front."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1109\/MMUL.2012.24","article-title":"Microsoft kinect sensor and its effect","volume":"19","author":"Zhang","year":"2012","journal-title":"IEEE Multimed."},{"key":"ref_11","unstructured":"Raschka, S. (2015). Python Machine Learning, Packt Publishing Ltd."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1535","DOI":"10.1007\/s00371-017-1429-y","article-title":"Spatial pyramid face feature representation and weighted dissimilarity matching for improved face recognition","volume":"34","author":"Choi","year":"2018","journal-title":"Vis. Comput."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"391","DOI":"10.1007\/s00371-019-01627-4","article-title":"Facial expression recognition with convolutional neural networks via a new face cropping and rotation strategy","volume":"36","author":"Li","year":"2020","journal-title":"Vis. Comput."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"951","DOI":"10.1007\/s00371-019-01705-7","article-title":"4D facial expression recognition using multimodal time series analysis of geometric landmark-based deformations","volume":"36","author":"Zarbakhsh","year":"2020","journal-title":"Vis. Comput."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Kubat, M. (2017). An Introduction to Machine Learning, Springer.","DOI":"10.1007\/978-3-319-63913-0"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Aguileta, A.A., Brena, R.F., Mayora, O., Molino-Minero-Re, E., and Trejo, L.A. (2019). Multi-Sensor Fusion for Activity Recognition\u2014A Survey. Sensors, 19.","DOI":"10.3390\/s19173808"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Aguileta, A.A., Brena, R.F., Mayora, O., Molino-Minero-Re, E., and Trejo, L.A. (2019). Virtual Sensors for Optimal Integration of Human Activity Data. Sensors, 19.","DOI":"10.3390\/s19092017"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"369","DOI":"10.1017\/S0142716400009632","article-title":"The acquisition of conditionals in American Sign Language: Grammaticized facial expressions","volume":"11","author":"Reilly","year":"1990","journal-title":"Appl. Psycholinguist."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"307","DOI":"10.1177\/00238309990420020801","article-title":"Neuropsychological Studies of Linguistic and Affective Facial Expressions in Deaf Signers","volume":"42","author":"Corina","year":"1999","journal-title":"Lang. Speech"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Uddin, M. (2015, January 15\u201318). An Ada-Random Forests based grammatical facial expressions recognition approach. Proceedings of the 2015 International Conference on Informatics, Electronics and Vision (ICIEV), Fukuoka, Japan.","DOI":"10.1109\/ICIEV.2015.7333976"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1007\/s00779-012-0615-1","article-title":"Non-manual cues in automatic sign language recognition","volume":"18","author":"Caridakis","year":"2014","journal-title":"Pers. Ubiquitous Comput."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1145\/2444800.2444804","article-title":"Models of linguistic facial expressions for American Sign Language animation","volume":"105","author":"Kacorri","year":"2013","journal-title":"ACM SIGACCESS Access. Comput."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"2022","DOI":"10.1109\/TPAMI.2010.28","article-title":"Features versus context: An approach for precise and detailed detection and delineation of faces and facial features","volume":"32","author":"Ding","year":"2010","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Von Agris, U., Knorr, M., and Kraiss, K.F. (2008, January 17\u201319). The significance of facial features for automatic sign language recognition. Proceedings of the 8th IEEE International Conference on Automatic Face & Gesture Recognition, Amsterdam, The Netherlands.","DOI":"10.1109\/AFGR.2008.4813472"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"519","DOI":"10.1134\/S1054661812040062","article-title":"Local Binary Pattern based features for sign language recognition","volume":"22","year":"2012","journal-title":"Pattern Recognit. Image Anal."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Ari, I., Uyar, A., and Akarun, L. (2008, January 27\u201329). Facial feature tracking and expression recognition for sign language. Proceedings of the 23rd International Symposium on Computer and Information Sciences, Istanbul, Turkey.","DOI":"10.1109\/ISCIS.2008.4717948"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"345","DOI":"10.1007\/s00530-010-0182-0","article-title":"Multimodal fusion for multimedia analysis: A survey","volume":"16","author":"Atrey","year":"2010","journal-title":"Multimed. Syst."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1109\/MASSP.1986.1165342","article-title":"An introduction to hidden Markov models","volume":"3","author":"Rabiner","year":"1986","journal-title":"IEEE Assp Mag."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1023\/A:1009715923555","article-title":"A tutorial on support vector machines for pattern recognition","volume":"2","author":"Burges","year":"1998","journal-title":"Data Min. Knowl. Discov."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1877","DOI":"10.1016\/j.patcog.2011.10.026","article-title":"Facial expressions in American sign language: Tracking and recognition","volume":"45","author":"Nguyen","year":"2012","journal-title":"Pattern Recognit."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1016\/j.ins.2017.10.046","article-title":"Independent Bayesian classifier combination based sign language recognition using facial expression","volume":"428","author":"Kumar","year":"2018","journal-title":"Inf. Sci."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Simpson, E., Roberts, S., Psorakis, I., and Smith, A. (2013). Dynamic bayesian combination of multiple imperfect classifiers. Decision Making and Imperfection, Springer.","DOI":"10.1007\/978-3-642-36406-8_1"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Da Silva, E.P., Costa, P.D.P., Kumada, K.M.O., De Martino, J.M., and Florentino, G.A. (2020, January 23\u201328). Recognition of Affective and Grammatical Facial Expressions: A Study for Brazilian Sign Language. Proceedings of the Computer Vision\u2014ECCV 2020 Workshops, Glasgow, UK.","DOI":"10.1007\/978-3-030-66096-3_16"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Aggarwal, C.C. (2018). Neural Networks and Deep Learning, Springer.","DOI":"10.1007\/978-3-319-94463-0"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1235","DOI":"10.1162\/neco_a_01199","article-title":"A review of recurrent neural networks: LSTM cells and network architectures","volume":"31","author":"Yu","year":"2019","journal-title":"Neural Comput."},{"key":"ref_36","unstructured":"Neidle, C., Michael, N., Nash, J., Metaxas, D., Bahan, I., Cook, L., Duffy, Q., and Lee, R. (2009, January 20\u201331). A method for recognition of grammatically significant head movements and facial expressions, developed through use of a linguistically annotated video corpus. Proceedings of the 21st ESSLLI Workshop on Formal Approaches to Sign Languages, Bordeaux, France."},{"key":"ref_37","unstructured":"De Almeida Freitas, F., Peres, S.M., de Moraes Lima, C.A., and Barbosa, F.V. (2014, January 21\u201323). Grammatical facial expressions recognition with machine learning. Proceedings of the Twenty-Seventh International Flairs Conference, Pensacola Beach, FL, USA."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"De Quadros, R.M., and Karnopp, L.B. (2004). L\u00edngua de Sinais Brasileira-Estudos Ling\u00fc\u00edsticos, Artmed Editora.","DOI":"10.18309\/anp.v1i16.560"},{"key":"ref_39","unstructured":"Brito, L.F. (1990). Uma Abordagem Fonol\u00f3gica dos Sinais da LSCB."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"683701","DOI":"10.1155\/2015\/683701","article-title":"A survey on multisensor fusion and consensus filtering for sensor networks","volume":"2015","author":"Li","year":"2015","journal-title":"Discret. Dyn. Nat. Soc."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Wang, T., Wang, X., and Hong, M. (2018). Gas Leak Location Detection Based on Data Fusion with Time Difference of Arrival and Energy Decay Using an Ultrasonic Sensor Array. Sensors, 18.","DOI":"10.3390\/s18092985"},{"key":"ref_42","unstructured":"Liggins, M.E., Hall, D.L., and Llinas, J. (2009). Handbook of Multisensor Data Fusion: Theory and Practice, CRC Press."},{"key":"ref_43","unstructured":"Zhou, Z.H. (2019). Ensemble Methods: Foundations and Algorithms, Chapman and Hall\/CRC."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Bhuvan, M.S., Rao, D.V., Jain, S., Ashwin, T.S., Guddetti, R.M.R., and Kulgod, S.P. (2016, January 9\u201311). Detection and analysis model for grammatical facial expressions in sign language. Proceedings of the 2016 IEEE Region 10 Symposium (TENSYMP), Bali, Indonesia.","DOI":"10.1109\/TENCONSpring.2016.7519396"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random Forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach. Learn."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1006\/jcss.1997.1504","article-title":"A decision-theoretic generalization of on-line learning and an application to boosting","volume":"55","author":"Freund","year":"1997","journal-title":"J. Comput. Syst. Sci."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"861","DOI":"10.1016\/j.patrec.2005.10.010","article-title":"An introduction to ROC analysis","volume":"27","author":"Fawcett","year":"2006","journal-title":"Pattern Recognit. Lett."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Bramer, M. (2016). Measuring the Performance of a Classifier. Principles of Data Mining, Springer. Chapter 12.","DOI":"10.1007\/978-1-4471-7307-6_12"},{"key":"ref_49","unstructured":"Arai, K., Kapoor, S., and Bhatia, R. (2019). Gramatical Facial Expression Recognition with Artificial Intelligence Tools. Intelligent Computing, Springer."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"281","DOI":"10.1109\/72.661123","article-title":"Morphological associative memories","volume":"9","author":"Ritter","year":"1998","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Gafar, M.G. (2019, January 2\u20134). Grammatical Facial Expression Recognition Basing on a Hybrid of Fuzzy Rough Ant Colony Optimization and Nearest Neighbor Classifier. Proceedings of the International Conference on Innovative Trends in Computer Engineering (ITCE), Aswan, Egypt.","DOI":"10.1109\/ITCE.2019.8646649"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"2134","DOI":"10.1016\/j.fss.2007.04.023","article-title":"Fuzzy-rough nearest neighbor algorithms in classification","volume":"158","author":"Sarkar","year":"2007","journal-title":"Fuzzy Sets Syst."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Chan, C.C., Grzymala-Busse, J.W., and Ziarko, W.P. (2008). A New Approach to Fuzzy-Rough Nearest Neighbour Classification. Rough Sets and Current Trends in Computing, Springer.","DOI":"10.1007\/978-3-540-88425-5"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Aha, D.W. (1997). Editorial. Lazy Learning, Springer.","DOI":"10.1007\/978-94-017-2053-3"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1016\/j.fss.2004.07.014","article-title":"Fuzzy-rough data reduction with ant colony optimization","volume":"149","author":"Jensen","year":"2005","journal-title":"Fuzzy Sets Syst."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Greco, S., Hata, Y., Hirano, S., Inuiguchi, M., Miyamoto, S., Nguyen, H.S., and S\u0142owi\u0144ski, R. (2006). Webpage Classification with ACO-Enhanced Fuzzy-Rough Feature Selection. Rough Sets and Current Trends in Computing, Springer.","DOI":"10.1007\/11908029"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"824","DOI":"10.1109\/TFUZZ.2008.924209","article-title":"New Approaches to Fuzzy-Rough Feature Selection","volume":"17","author":"Jensen","year":"2009","journal-title":"IEEE Trans. Fuzzy Syst."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"469","DOI":"10.1016\/S0165-0114(03)00021-6","article-title":"Fuzzy\u2013rough attribute reduction with application to web categorization","volume":"141","author":"Jensen","year":"2004","journal-title":"Fuzzy Sets Syst."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1109\/MCI.2006.329691","article-title":"Ant colony optimization","volume":"1","author":"Dorigo","year":"2006","journal-title":"IEEE Comput. Intell. Mag."},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"\u00dcnal, M., and Ak, A. (2013). Ant Colony Optimization (ACO). Optimization of PID Controllers Using Ant Colony and Genetic Algorithms, Springer.","DOI":"10.1007\/978-3-642-32900-5"},{"key":"ref_61","unstructured":"Quinlan, J.R. (2014). C4. 5: Programs for Machine Learning, Elsevier."},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"580","DOI":"10.1109\/TSMC.1985.6313426","article-title":"A fuzzy K-nearest neighbor algorithm","volume":"4","author":"Keller","year":"1985","journal-title":"IEEE Trans. Syst. Man Cybern."},{"key":"ref_63","doi-asserted-by":"crossref","unstructured":"Bramer, M. (2016). Classifying Streaming Data II: Time-Dependent Data. Principles of Data Mining, Springer. Chapter 22.","DOI":"10.1007\/978-1-4471-7307-6"},{"key":"ref_64","unstructured":"Cardoso, M.E.D.A., Freitas, F.D.A., Barbosa, F.V., Lima, C.A.D.M., Peres, S.M., and Hung, P.C. (2020, January 7\u201310). Automatic segmentation of grammatical facial expressions in sign language: Towards an inclusive communication experience. Proceedings of the 53rd Hawaii International Conference on System Science, Maui, HI, USA."},{"key":"ref_65","unstructured":"Dua, D., and Graff, C. (2022, January 27). UCI Machine Learning Repository, University of California, Irvine, School of Information and Computer Sciences. Available online: http:\/\/archive.ics.uci.edu\/ml."},{"key":"ref_66","first-page":"1","article-title":"Calculating the area and centroid of a polygon","volume":"7","author":"Bourke","year":"1988","journal-title":"Swinburne Univ. Technol."},{"key":"ref_67","unstructured":"Horej\u0161, J. (2010, January 20). Shape Analysis Using Global Shape Measures. Proceedings of the 18th Annual Conference Proceedings, Technical Computing, Bratislava, Slovakia."},{"key":"ref_68","unstructured":"Tan, P.N., Steinbach, M., and Kumar, V. (2005). Introduction to Data Mining, Pearson Addison-Wesley."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/11\/4206\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:22:57Z","timestamp":1760138577000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/11\/4206"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,31]]},"references-count":68,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2022,6]]}},"alternative-id":["s22114206"],"URL":"https:\/\/doi.org\/10.3390\/s22114206","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,5,31]]}}}