{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,24]],"date-time":"2026-03-24T16:26:22Z","timestamp":1774369582771,"version":"3.50.1"},"reference-count":64,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2018,2,1]],"date-time":"2018-02-01T00:00:00Z","timestamp":1517443200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Extensive possibilities of applications have made emotion recognition ineluctable and challenging in the field of computer science. The use of non-verbal cues such as gestures, body movement, and facial expressions convey the feeling and the feedback to the user. This discipline of Human\u2013Computer Interaction places reliance on the algorithmic robustness and the sensitivity of the sensor to ameliorate the recognition. Sensors play a significant role in accurate detection by providing a very high-quality input, hence increasing the efficiency and the reliability of the system. Automatic recognition of human emotions would help in teaching social intelligence in the machines. This paper presents a brief study of the various approaches and the techniques of emotion recognition. The survey covers a succinct review of the databases that are considered as data sets for algorithms detecting the emotions by facial expressions. Later, mixed reality device Microsoft HoloLens (MHL) is introduced for observing emotion recognition in Augmented Reality (AR). A brief introduction of its sensors, their application in emotion recognition and some preliminary results of emotion recognition using MHL are presented. The paper then concludes by comparing results of emotion recognition by the MHL and a regular webcam.<\/jats:p>","DOI":"10.3390\/s18020416","type":"journal-article","created":{"date-parts":[[2018,2,2]],"date-time":"2018-02-02T04:20:50Z","timestamp":1517545250000},"page":"416","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":152,"title":["Facial Emotion Recognition: A Survey and Real-World User Experiences in Mixed Reality"],"prefix":"10.3390","volume":"18","author":[{"given":"Dhwani","family":"Mehta","sequence":"first","affiliation":[{"name":"EECS Department, The University of Toledo, Toledo, OH 43606, USA"}]},{"given":"Mohammad","family":"Siddiqui","sequence":"additional","affiliation":[{"name":"EECS Department, The University of Toledo, Toledo, OH 43606, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4719-4941","authenticated-orcid":false,"given":"Ahmad","family":"Javaid","sequence":"additional","affiliation":[{"name":"EECS Department, The University of Toledo, Toledo, OH 43606, USA"}]}],"member":"1968","published-online":{"date-parts":[[2018,2,1]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Ekman, P., and Friesen, W.V. (1977). Facial Action Coding System, Weidenfeld and Nicolson. [2nd ed.].","DOI":"10.1037\/t27734-000"},{"key":"ref_2","unstructured":"Zhang, C., and Zhang, Z. (2010). A Survey of Recent Advances in Face Detection, Microsoft Corporation. TechReport, No. MSR-TR-2010-66."},{"key":"ref_3","unstructured":"Ekman, P., Friesen, W., and Hager, J. (2002). Facial Action Coding System: The Manual on CD ROM, A Human Face."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Anil, J., and Suresh, L.P. (2016, January 18\u201319). Literature survey on face and face expression recognition. Proceedings of the 2016 International Conference on Circuit, Power and Computing Technologies (ICCPCT), Nagercoil, India.","DOI":"10.1109\/ICCPCT.2016.7530173"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"486","DOI":"10.1016\/j.procs.2015.08.011","article-title":"Facial expression recognition: A survey","volume":"58","author":"Kumari","year":"2015","journal-title":"Procedia Comput. Sci."},{"key":"ref_6","unstructured":"Cohn, J.F., Zlochower, A.J., Lien, J.J., and Kanade, T. (1998, January 14\u201316). Feature-point tracking by optical flow discriminates subtle differences in facial expression. Proceedings of the Third IEEE International Conference on Automatic Face and Gesture Recognition, Nara, Japan."},{"key":"ref_7","unstructured":"(2018, January 05). The Meta 2: Made for AR App Development. Available online: http:\/\/www.metavision.com\/."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1548","DOI":"10.1109\/TPAMI.2016.2515606","article-title":"Survey on RGB, 3D, thermal, and multimodal approaches for facial expression recognition: History, trends, and affect-related applications","volume":"38","author":"Corneanu","year":"2016","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_9","unstructured":"Vural, E., \u00c7etin, M., Er\u00e7il, A., Littlewort, G., Bartlett, M., and Movellan, J. (2008, January 13\u201314). Automated drowsiness detection for improved driving safety. Proceedings of the 4th International conference on Automotive Technologies, Istanbul, Turkey."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"555","DOI":"10.1016\/S0893-6080(03)00115-1","article-title":"Subject independent facial expression recognition with robust face detection using a convolutional neural network","volume":"16","author":"Matsugu","year":"2003","journal-title":"Neural Netw."},{"key":"ref_11","unstructured":"Viola, P., and Jones, M. (2001;, January 8\u201314). Rapid object detection using a boosted cascade of simple features. Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2001), Kauai, HI, USA."},{"key":"ref_12","first-page":"1197","article-title":"Synergistic face detection and pose estimation with energy-based models","volume":"8","author":"Osadchy","year":"2007","journal-title":"J. Mach. Learn. Res."},{"key":"ref_13","unstructured":"Dalal, N., and Triggs, B. (2005, January 20\u201325). Histograms of oriented gradients for human detection. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, (CVPR 2005), San Diego, CA, USA."},{"key":"ref_14","unstructured":"Fasel, B. (2002, January 11\u201315). Robust face analysis using convolutional neural networks. Proceedings of the 16th International Conference on Pattern Recognition, Quebec City, QC, Canada."},{"key":"ref_15","unstructured":"Mandal, T., Majumdar, A., and Wu, Q.J. (2007, January 22\u201324). Face recognition by curvelet based feature extraction. Proceedings of the International Conference Image Analysis and Recognition, Montreal, QC, Canada."},{"key":"ref_16","unstructured":"Deans, S.R. (2007). The Radon Transform and Some of Its Applications, Courier Corporation."},{"key":"ref_17","first-page":"30","article-title":"Automatic Facial Expression Recognition Using 3D Faces","volume":"3","author":"Li","year":"2011","journal-title":"Int. J. Eng. Res. Innov."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Dhall, A., Goecke, R., Joshi, J., Hoey, J., and Gedeon, T. (2016, January 12\u201316). Emotiw 2016: Video and group-level emotion recognition challenges. Proceedings of the 18th ACM International Conference on Multimodal Interaction, Tokyo, Japan.","DOI":"10.1145\/2993148.2997638"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"2588","DOI":"10.1016\/j.patcog.2011.03.013","article-title":"Human face recognition based on multidimensional PCA and extreme learning machine","volume":"44","author":"Mohammed","year":"2011","journal-title":"Pattern Recognit."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1740","DOI":"10.1109\/TIP.2012.2235848","article-title":"Local directional number pattern for face analysis: Face and expression recognition","volume":"22","author":"Rivera","year":"2013","journal-title":"IEEE Trans. Image Process."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1109\/TSMCA.2009.2033031","article-title":"FARO: Face recognition against occlusions and expression variations","volume":"40","author":"Nappi","year":"2010","journal-title":"IEEE Trans. Syst. Man Cybern. Part A Syst. Hum."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Alex, A.T., Asari, V.K., and Mathew, A. (2012, January 14\u201317). Gradient feature matching for expression invariant face recognition using single reference image. Proceedings of the 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Seoul, Korea.","DOI":"10.1109\/ICSMC.2012.6377834"},{"key":"ref_23","unstructured":"Kanade, T., Cohn, J.F., and Tian, Y. (2000, January 28\u201330). Comprehensive database for facial expression analysis. Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition, Grenoble, France."},{"key":"ref_24","first-page":"24","article-title":"A survey on human emotion recognition approaches, databases and applications","volume":"14","author":"Vinola","year":"2015","journal-title":"Electron. Lett. Comput. Vis. Image Anal."},{"key":"ref_25","unstructured":"Yin, L., Wei, X., Sun, Y., Wang, J., and Rosato, M.J. (2006, January 10\u201312). A 3D facial expression database for facial behavior research. Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition, (FGR 2006), Southampton, UK."},{"key":"ref_26","unstructured":"Martinez, A.M. (1998). The AR Face Database, The Ohio State University. CVC Technical Report."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1023\/B:VISI.0000029666.37597.d3","article-title":"Active appearance models revisited","volume":"60","author":"Matthews","year":"2004","journal-title":"Int. J. Comput. Vis."},{"key":"ref_28","unstructured":"Phillips, P. (2004). The Facial Recognition Technology (FERET) Database, NIST."},{"key":"ref_29","unstructured":"(2017, November 29). The Essex Faces94 Database. Available online: http:\/\/cswww.essex.ac.uk\/mv\/allfaces\/."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1117\/12.263232","article-title":"Real-Time detection of human faces in uncontrolled environments","volume":"Volume 3024","author":"Nefian","year":"1997","journal-title":"Proceedings of the SPIE Conference on Visual Communications and Image Processing"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Graham, D.B., and Allinson, N.M. (1998). Characterising virtual eigensignatures for general purpose face recognition. Face Recognition, Springer.","DOI":"10.1007\/978-3-642-72201-1_25"},{"key":"ref_32","unstructured":"Samaria, F.S., and Harter, A.C. (1994, January 5\u20137). Parameterisation of a stochastic model for human face identification. Proceedings of the Second IEEE Workshop on Applications of Computer Vision, Sarasota, FL, USA."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1450","DOI":"10.1109\/TNN.2002.804287","article-title":"Face recognition by independent component analysis","volume":"13","author":"Bartlett","year":"2002","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Ebrahimi Kahou, S., Michalski, V., Konda, K., Memisevic, R., and Pal, C. (2015). Recurrent Neural Networks for Emotion Recognition in Video. ICMI \u201815, Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, ACM.","DOI":"10.1145\/2818346.2830596"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Kahou, S.E., Pal, C., Bouthillier, X., Froumenty, P., G\u00fcl\u00e7ehre, \u00c7., Memisevic, R., Vincent, P., Courville, A., Bengio, Y., and Ferrari, R.C. (2013, January 9\u201313). Combining modality specific deep neural networks for emotion recognition in video. Proceedings of the 15th ACM on International Conference On Multimodal Interaction, Sydney, Australia.","DOI":"10.1145\/2522848.2531745"},{"key":"ref_36","unstructured":"Krizhevsky, A., and Hinton, G. (2009). Learning Multiple Layers of Features from Tiny Images, University of Toronto. Technical Report."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Yu, Z., and Zhang, C. (2015, January 9\u201313). Image based static facial expression recognition with multiple deep network learning. Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, Seattle, WA, USA.","DOI":"10.1145\/2818346.2830595"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Dhall, A., Goecke, R., Lucey, S., and Gedeon, T. (2011, January 6\u201313). Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark. Proceedings of the 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain.","DOI":"10.1109\/ICCVW.2011.6130508"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"803","DOI":"10.1016\/j.imavis.2008.08.005","article-title":"Facial expression recognition based on local binary patterns: A comprehensive study","volume":"27","author":"Shan","year":"2009","journal-title":"Image Vis. Comput."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1016\/j.cviu.2015.07.007","article-title":"Adaptive facial point detection and emotion recognition for a humanoid robot","volume":"140","author":"Zhang","year":"2015","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_41","unstructured":"Grgic, M., and Delac, K. (2018, January 02). Face Recognition Homepage. Zagreb, Croatia. Available online: www.face-rec.org\/databases."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1109\/T-AFFC.2011.13","article-title":"Facial expression recognition using facial movement features","volume":"2","author":"Zhang","year":"2011","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1109\/TAFFC.2014.2330580","article-title":"An automatic framework for textured 3D video-based facial expression recognition","volume":"5","author":"Hayat","year":"2014","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_44","first-page":"163","article-title":"Recognition of facial expressions using local binary patterns of important facial parts","volume":"7","author":"Hablani","year":"2013","journal-title":"Int. J. Image Process."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Li, Z., Imai, J.-i., and Kaneko, M. (2009, January 11\u201314). Facial-component-based bag of words and phog descriptor for facial expression recognition. Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, (SMC 2009), San Antonio, TX, USA.","DOI":"10.1109\/ICSMC.2009.5346254"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"340","DOI":"10.1109\/TAFFC.2014.2346515","article-title":"Intra-class variation reduction using training expression images for sparse representation based facial expression recognition","volume":"5","author":"Lee","year":"2014","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1109\/TAFFC.2014.2304712","article-title":"Multi-view facial expression recognition based on group sparse reduced-rank regression","volume":"5","author":"Zheng","year":"2014","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"467","DOI":"10.1016\/j.engappai.2012.09.002","article-title":"Facial expression recognition using tracked facial actions: Classifier performance analysis","volume":"26","author":"Dornaika","year":"2013","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1109\/TAFFC.2014.2317711","article-title":"Fully automated recognition of spontaneous facial expressions in videos using random forest classifiers","volume":"5","author":"Levine","year":"2014","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"5160","DOI":"10.1016\/j.eswa.2013.03.016","article-title":"Intelligent facial emotion recognition and semantic-based topic detection for a humanoid robot","volume":"40","author":"Zhang","year":"2013","journal-title":"Expert Syst. Appl."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Wu, T., Bartlett, M.S., and Movellan, J.R. (2010, January 13\u201318). Facial expression recognition using gabor motion energy filters. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), San Francisco, CA, USA.","DOI":"10.1109\/CVPRW.2010.5543267"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Jain, S., Hu, C., and Aggarwal, J.K. (2011, January 6\u201313). Facial expression recognition with temporal modeling of shapes. Proceedings of the 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain.","DOI":"10.1109\/ICCVW.2011.6130446"},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., and Matthews, I. (2010, January 13\u201318). The extended cohn-kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), San Francisco, CA, USA.","DOI":"10.1109\/CVPRW.2010.5543262"},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"433","DOI":"10.1109\/TSMCB.2005.859075","article-title":"Dynamics of facial expression: Recognition of facial actions and their temporal segments from face profile image sequences","volume":"36","author":"Pantic","year":"2006","journal-title":"IEEE Trans. Syst. Man, and Cybern. Part B Cybern."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"807","DOI":"10.1016\/j.imavis.2009.08.002","article-title":"Multi-pie","volume":"28","author":"Gross","year":"2010","journal-title":"Image Vis. Comput."},{"key":"ref_56","unstructured":"Dhall, A., Goecke, R., Lucey, S., and Gedeon, T. (2011). Acted Facial Expressions in the Wild Database, Australian National University. Technical Report TR-CS-11."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1109\/T-AFFC.2011.20","article-title":"The semaine database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent","volume":"3","author":"McKeown","year":"2012","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_58","unstructured":"(2018, January 05). Dataset 02: IRIS Thermal\/Visible Face Database. Available online: http:\/\/vcipl-okstate.org\/pbvs\/bench\/."},{"key":"ref_59","unstructured":"(2018, January 05). NIST Dataset, Available online: https:\/\/www.nist.gov\/srd\/nist-special-database-19."},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"682","DOI":"10.1109\/TMM.2010.2060716","article-title":"A natural visible and infrared facial expression database for expression recognition and emotion inference","volume":"12","author":"Wang","year":"2010","journal-title":"IEEE Trans. Multimed."},{"key":"ref_61","unstructured":"Nguyen, H., Kotani, K., Chen, F., and Le, B. (November, January 28). A thermal facial emotion database and its analysis. Proceedings of the Pacific-Rim Symposium on Image and Video Technology, Guanajuato, Mexico."},{"key":"ref_62","doi-asserted-by":"crossref","unstructured":"Savran, A., Aly\u00fcz, N., Dibeklio\u011flu, H., \u00c7eliktutan, O., G\u00f6kberk, B., Sankur, B., and Akarun, L. (2008, January 7\u20139). Bosphorus database for 3D face analysis. Proceedings of the European Workshop on Biometrics and Identity Management, Roskilde, Denmark.","DOI":"10.1007\/978-3-540-89991-4_6"},{"key":"ref_63","unstructured":"(2018, January 05). HoloLens Hardware Details. Available online: https:\/\/developer.microsoft.com\/en-us\/windows\/mixed-reality\/hololens_hardware_details#sensors."},{"key":"ref_64","unstructured":"(2018, January 05). How to Analyze Videos in Real-Time. Available online: https:\/\/docs.microsoft.com\/en-us\/azure\/cognitive-services\/emotion\/emotion-api-how-to-topics\/howtoanalyzevideo_emotion."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/18\/2\/416\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T14:53:20Z","timestamp":1760194400000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/18\/2\/416"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,2,1]]},"references-count":64,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2018,2]]}},"alternative-id":["s18020416"],"URL":"https:\/\/doi.org\/10.3390\/s18020416","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,2,1]]}}}