{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,14]],"date-time":"2026-03-14T04:59:16Z","timestamp":1773464356848,"version":"3.50.1"},"reference-count":35,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2024,6,4]],"date-time":"2024-06-04T00:00:00Z","timestamp":1717459200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000780","name":"European Commission","doi-asserted-by":"publisher","award":["101021714"],"award-info":[{"award-number":["101021714"]}],"id":[{"id":"10.13039\/501100000780","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MTI"],"abstract":"<jats:p>This paper researches the classification of human emotions in a virtual reality (VR) context by analysing psychophysiological signals and facial expressions. Key objectives include exploring emotion categorisation models, identifying critical human signals for assessing emotions, and evaluating the accuracy of these signals in VR environments. A systematic literature review was performed through peer-reviewed articles, forming the basis for our methodologies. The integration of various emotion classifiers employs a \u2018late fusion\u2019 technique due to varying accuracies among classifiers. Notably, facial expression analysis faces challenges from VR equipment occluding crucial facial regions like the eyes, which significantly impacts emotion recognition accuracy. A weighted averaging system prioritises the psychophysiological classifier over the facial recognition classifiers due to its higher accuracy. Findings suggest that while combined techniques are promising, they struggle with mixed emotional states as well as with fear and trust emotions. The research underscores the potential and limitations of current technologies, recommending enhanced algorithms for effective interpretation of complex emotional expressions in VR. The study provides a groundwork for future advancements, aiming to refine emotion recognition systems through systematic data collection and algorithm optimisation.<\/jats:p>","DOI":"10.3390\/mti8060047","type":"journal-article","created":{"date-parts":[[2024,6,4]],"date-time":"2024-06-04T07:39:49Z","timestamp":1717486789000},"page":"47","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Exploring Human Emotions: A Virtual Reality-Based Experimental Approach Integrating Physiological and Facial Analysis"],"prefix":"10.3390","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2399-2757","authenticated-orcid":false,"given":"Leire","family":"Bastida","sequence":"first","affiliation":[{"name":"TECNALIA Research and Basque Research and Technology Alliance (BRTA), 48160 Derio, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6027-8931","authenticated-orcid":false,"given":"Sara","family":"Sillaurren","sequence":"additional","affiliation":[{"name":"TECNALIA, Basque Research and Technology Alliance (BRTA), 01510 Vitoria, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6481-7536","authenticated-orcid":false,"given":"Erlantz","family":"Loizaga","sequence":"additional","affiliation":[{"name":"TECNALIA Research and Basque Research and Technology Alliance (BRTA), 48160 Derio, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0355-3829","authenticated-orcid":false,"given":"Eneko","family":"Tom\u00e9","sequence":"additional","affiliation":[{"name":"TECNALIA Research and Basque Research and Technology Alliance (BRTA), 48160 Derio, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ana","family":"Moya","sequence":"additional","affiliation":[{"name":"TECNALIA Research and Basque Research and Technology Alliance (BRTA), 48160 Derio, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,6,4]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Ortony, A., Clore, G.L., and Collins, A. (2022). The Cognitive Structure of Emotions, Cambridge University Press.","DOI":"10.1017\/9781108934053"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1146\/annurev.psych.60.110707.163539","article-title":"Emotion theory and research: Highlights, unanswered questions, and emerging issues","volume":"60","author":"Izard","year":"2009","journal-title":"Annu. Rev. Psychol."},{"key":"ref_3","first-page":"10","article-title":"The Coherent Heart Heart\u2013Brain Interactions, Psychophysiological Coherence, and the Emergence of System-Wide Order","volume":"5","author":"McCraty","year":"2009","journal-title":"Integral Rev."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1016\/j.entcs.2019.04.009","article-title":"Emotion Recognition from Physiological Signal Analysis: A Review","volume":"343","author":"Egger","year":"2019","journal-title":"Electron. Notes Theor. Comput. Sci."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1093014","DOI":"10.3389\/fpsyg.2023.1093014","article-title":"The Reality of Virtual Reality","volume":"14","author":"Kisker","year":"2023","journal-title":"Front. Psychol."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"2582","DOI":"10.1109\/TAFFC.2023.3265008","article-title":"Emotion arousal assessment based on multimodal physiological signals for game users","volume":"14","author":"Li","year":"2023","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Shu, L. (2018). A review of emotion recognition using physiological signals. Sensors, 18.","DOI":"10.3390\/s18072074"},{"key":"ref_8","first-page":"651","article-title":"Emotion Models: A Review","volume":"10","author":"Sreeja","year":"2017","journal-title":"Int. J. Control. Theory Appl."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Baig, M., and Kavakli, M. (2019). A Survey on Psycho-Physiological Analysis & Measurement Methods in Multimodal Systems. Multimodal Technol. Interact., 3.","DOI":"10.3390\/mti3020037"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Dzedzickis, A., Kaklauskas, A., and Bu\u010dinskas, V. (2020). Human Emotion Recognition: Review of Sensors and Methods. Sensors, 20.","DOI":"10.3390\/s20030592"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"384","DOI":"10.1037\/0003-066X.48.4.384","article-title":"Facial expression and emotion","volume":"48","author":"Ekman","year":"1993","journal-title":"Am. Psychol."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"344","DOI":"10.1511\/2001.28.344","article-title":"The Nature of Emotions","volume":"89","author":"Plutchik","year":"2001","journal-title":"Am. Sci."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"493","DOI":"10.1037\/0022-3514.57.3.493","article-title":"Affect grid: A single-item scale of pleasure and arousal","volume":"57","author":"Russel","year":"1989","journal-title":"J. Pers. Soc. Psychol."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1007\/BF02229025","article-title":"Comparison of the PAD and PANAS as models for describing emotions and for differentiating anxiety from depression","volume":"19","author":"Mehrabian","year":"1997","journal-title":"J. Psychopathol. Behav. Assess."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"2159","DOI":"10.1007\/s11042-015-3119-y","article-title":"Affect representation and recognition in 3D continuous valence\u2013arousal\u2013dominance space","volume":"76","author":"Verma","year":"2017","journal-title":"Multimed. Tools Appl."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1109\/TITS.2005.848368","article-title":"Detecting stress during real-world driving tasks using physiological sensors","volume":"6","author":"Healey","year":"2005","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"394","DOI":"10.1016\/j.biopsycho.2010.03.010","article-title":"Autonomic nervous system activity in emotion: A review","volume":"84","author":"Kreibig","year":"2010","journal-title":"Biol. Psychol."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"417","DOI":"10.1109\/TAFFC.2017.2712143","article-title":"Identifying Stable Patterns over Time for Emotion Recognition from EEG","volume":"10","author":"Zheng","year":"2016","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"84","DOI":"10.3389\/fncom.2021.758212","article-title":"Review on Emotion Recognition Based on Electroencephalography","volume":"15","author":"Liu","year":"2021","journal-title":"Front. Comput. Neurosci."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"140990","DOI":"10.1109\/ACCESS.2019.2944001","article-title":"A Review, Current Challenges, and Future Possibilities on Emotion Recognition Using Machine Learning and Physiological Signals","volume":"7","author":"Bota","year":"2019","journal-title":"IEEE Access"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Hasnul, M., Abd Aziz, N., Alelyani, S., Mohana, M., and Abd Aziz, A. (2021). Electrocardiogram-Based Emotion Recognition Systems and Their Applications in Healthcare\u2014A Review. Sensors, 21.","DOI":"10.3390\/s21155015"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1177\/1754073917749016","article-title":"Experimental Methods for Inducing Basic Emotions: A Qualitative Review","volume":"11","author":"Siedlecka","year":"2019","journal-title":"Emot. Rev."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Houshmand, B., and Khan, N. (2020, January 24\u201326). Facial Expression Recognition Under Partial Occlusion from Virtual Reality Headsets based on Transfer Learning. Proceedings of the IEEE Sixth International Conference on Multimedia Big Data, New Delhi, India.","DOI":"10.1109\/BigMM50055.2020.00020"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1085","DOI":"10.1037\/a0025737","article-title":"Emotion expression in body action and posture","volume":"12","author":"Dael","year":"2012","journal-title":"Emotion"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Chen, J., Hu, B., Wang, Y., Moore, P., Dai, Y., Feng, L., and Ding, Z. (2017). Subject-independent emotion recognition based on physiological signals: A three-stage decision method. BMC Med. Inform. Decis. Mak., 17.","DOI":"10.1186\/s12911-017-0562-x"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Ali, M., Al Machot, F., Haj Mosa, A., Jdeed, M., Al Machot, E., and Kyamakya, K. (2018). A Globally Generalized Emotion Recognition System Involving Different Physiological Signals. Sensors, 18.","DOI":"10.3390\/s18061905"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"768","DOI":"10.1037\/0022-3514.54.5.768","article-title":"Inhibiting and facilitating conditions of the human smile: A nonobtrusive test of the facial feedback hypothesis","volume":"54","author":"Strack","year":"1998","journal-title":"J. Personal. Soc. Psychol."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"966","DOI":"10.1037\/0022-3514.34.5.966","article-title":"Encoding and decoding of spontaneous and posed facial expressions","volume":"34","author":"Zuckerman","year":"1976","journal-title":"J. Pers. Soc. Psychol."},{"key":"ref_29","first-page":"1","article-title":"Facial expression of emotion: A comparison of posed expressions versus spontaneous expressions in an interpersonal communication setting","volume":"52","author":"Motley","year":"1988","journal-title":"West. J. Commun."},{"key":"ref_30","first-page":"113","article-title":"An Approach for Evaluation and Recognition of Facial Emotions Using EMG Signal","volume":"14","author":"Maity","year":"2024","journal-title":"Int. J. Sens. Wirel. Commun. Control"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"784834","DOI":"10.3389\/fpsyg.2021.784834","article-title":"Spontaneous facial expressions and micro-expressions coding: From brain to face","volume":"12","author":"Dong","year":"2022","journal-title":"Front. Psychol."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"2240005","DOI":"10.1142\/S0219467822400058","article-title":"A survey on various deep learning algorithms for an efficient facial expression recognition system","volume":"23","author":"Banerjee","year":"2023","journal-title":"Int. J. Image Graph."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"580287","DOI":"10.3389\/fpsyg.2020.580287","article-title":"Detection of Genuine and Posed Facial Expressions of Emotion: Databases and Methods","volume":"11","author":"Jia","year":"2021","journal-title":"Front. Front. Psychol."},{"key":"ref_34","unstructured":"(2024, April 13). Pixta. Available online: https:\/\/www.pixtastock.com\/."},{"key":"ref_35","unstructured":"(2024, April 13). Datatang. Available online: https:\/\/m.datatang.ai\/."}],"container-title":["Multimodal Technologies and Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2414-4088\/8\/6\/47\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T14:53:30Z","timestamp":1760108010000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2414-4088\/8\/6\/47"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,4]]},"references-count":35,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2024,6]]}},"alternative-id":["mti8060047"],"URL":"https:\/\/doi.org\/10.3390\/mti8060047","relation":{},"ISSN":["2414-4088"],"issn-type":[{"value":"2414-4088","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,4]]}}}