{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,9]],"date-time":"2026-09-09T14:19:32Z","timestamp":1788963572224,"version":"build-2803163510"},"reference-count":34,"publisher":"MDPI AG","issue":"18","license":[{"start":{"date-parts":[[2026,9,9]],"date-time":"2026-09-09T00:00:00Z","timestamp":1788912000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Immersive Virtual Reality (VR) environments are increasingly adopted in clinical psychology and stress-management contexts; however, their therapeutic effectiveness depends on the system\u2019s ability to understand and dynamically respond to users\u2019 affective states. Emotional regulation plays a key role in psychological resilience, directly influencing stress coping mechanisms, cognitive performance, and overall mental well-being. Despite recent advances, automatic recognition of scenario-associated affective conditions in VR remains challenging because head-mounted displays occlude facial features. This study proposes a VR-based serious game for affective training and regulation, in which users interact with goal-oriented scenarios targeting fear, anger, and joy. We introduce a multimodal affective computing model to objectively assess users\u2019 emotional responses by integrating electrocardiogram (ECG) signals and posture-based features extracted through computer vision. An early-fusion architecture combined with a Long Short-Term Memory (LSTM) network captures temporal dependencies in synchronized multimodal data. We established a controlled experimental framework using immersive VR scenarios, enabling the collection of synchronized physiological and behavioral data from a cohort of 20 healthy adult participants. The proposed model was evaluated under a strict Leave-One-Subject-Out (LOSO) cross-validation scheme across independent subjects, achieving a robust inter-subject accuracy of 78.94%\u00b110.77% and a global macro F1-score of 0.635, demonstrating strong generalization to entirely unseen users without data leakage. Furthermore, the system maintained an outstanding balance in detecting active emotional states (recall &gt; 80.0% for fear, anger, and joy). Additionally, subjective evaluations using the PANAS and SGU questionnaires confirmed the coherence between detected and perceived emotional states, as well as the system\u2019s high usability. The results suggest the potential viability of combining immersive environments and multimodal affective computing to explore the technical feasibility of adaptive frameworks that could eventually translate into healthcare contexts. This work may contribute to the development of intelligent digital health technologies by providing a foundation for responsive VR systems that can monitor emotional regulation and are fully aligned with sustainable well-being ecosystems (SDG 3: Good Health and Well-being and SDG 10: Reduced Inequalities).<\/jats:p>","DOI":"10.3390\/s26185726","type":"journal-article","created":{"date-parts":[[2026,9,9]],"date-time":"2026-09-09T14:02:48Z","timestamp":1788962568000},"page":"5726","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Emotion-Aware Virtual Reality Through Multimodal ECG and Postural Fusion"],"prefix":"10.3390","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4527-7137","authenticated-orcid":false,"given":"Juan","family":"Benavides","sequence":"first","affiliation":[{"name":"Departamento de Inform\u00e1tica y Ciencias de la Computaci\u00f3n, Escuela Polit\u00e9cnica Nacional, Quito 170525, Ecuador"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"conceptualization","vocabulary":"credit"},{"role":"formal-analysis","vocabulary":"credit"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"methodology","vocabulary":"credit"},{"role":"investigation","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"},{"role":"software","vocabulary":"credit"},{"role":"resources","vocabulary":"credit"},{"role":"visualization","vocabulary":"credit"},{"role":"validation","vocabulary":"credit"},{"role":"data-curation","vocabulary":"credit"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5720-1960","authenticated-orcid":false,"given":"Mayra","family":"Carri\u00f3n-Toro","sequence":"additional","affiliation":[{"name":"Departamento de Inform\u00e1tica y Ciencias de la Computaci\u00f3n, Escuela Polit\u00e9cnica Nacional, Quito 170525, Ecuador"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"conceptualization","vocabulary":"credit"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"},{"role":"validation","vocabulary":"credit"},{"role":"methodology","vocabulary":"credit"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9182-6865","authenticated-orcid":false,"given":"Cindy","family":"L\u00f3pez","sequence":"additional","affiliation":[{"name":"Departamento de Inform\u00e1tica y Ciencias de la Computaci\u00f3n, Escuela Polit\u00e9cnica Nacional, Quito 170525, Ecuador"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"supervision","vocabulary":"credit"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"},{"role":"methodology","vocabulary":"credit"},{"role":"conceptualization","vocabulary":"credit"},{"role":"validation","vocabulary":"credit"},{"role":"investigation","vocabulary":"credit"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6578-3013","authenticated-orcid":false,"given":"David","family":"Morales-Mart\u00ednez","sequence":"additional","affiliation":[{"name":"Departamento de Inform\u00e1tica y Ciencias de la Computaci\u00f3n, Escuela Polit\u00e9cnica Nacional, Quito 170525, Ecuador"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"},{"role":"investigation","vocabulary":"credit"},{"role":"data-curation","vocabulary":"credit"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7251-923X","authenticated-orcid":false,"given":"Marco","family":"Sant\u00f3rum","sequence":"additional","affiliation":[{"name":"Departamento de Inform\u00e1tica y Ciencias de la Computaci\u00f3n, Escuela Polit\u00e9cnica Nacional, Quito 170525, Ecuador"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"},{"role":"supervision","vocabulary":"credit"},{"role":"validation","vocabulary":"credit"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4210-0117","authenticated-orcid":false,"given":"Patricia","family":"Acosta-Vargas","sequence":"additional","affiliation":[{"name":"Intelligent and Interactive Systems Laboratory, Universidad de Las Am\u00e9ricas, Quito 170503, Ecuador"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"},{"role":"funding-acquisition","vocabulary":"credit"},{"role":"project-administration","vocabulary":"credit"},{"role":"supervision","vocabulary":"credit"},{"role":"validation","vocabulary":"credit"},{"role":"resources","vocabulary":"credit"},{"role":"investigation","vocabulary":"credit"}]}],"member":"1968","published-online":{"date-parts":[[2026,9,9]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"01022","DOI":"10.1051\/itmconf\/20246301022","article-title":"Virtual Reality and Emotional Responses: A Comprehensive Literature Review on Theories, Frameworks, and Research Gaps","volume":"Volume 63","author":"Hariyady","year":"2024","journal-title":"ITM Web of Conferences"},{"key":"ref_2","unstructured":"Riavy, R. (2022). Serious Games for Emotional and Cognitive Training: A Systematic Review. [Master\u2019s Thesis, Universitat Polit\u00e8cnica de Val\u00e8ncia]."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"178","DOI":"10.1016\/j.compedu.2015.11.003","article-title":"An update to the systematic literature review of empirical evidence of the impacts and outcomes of computer games and serious games","volume":"94","author":"Boyle","year":"2016","journal-title":"Comput. Educ."},{"key":"ref_4","first-page":"1","article-title":"Immersive virtual reality and learning: A meta-analysis","volume":"31","author":"Makransky","year":"2019","journal-title":"Educ. Psychol. Rev."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"103778","DOI":"10.1016\/j.compedu.2019.103778","article-title":"A systematic review of immersive virtual reality applications for higher education: Design elements, lessons learned, and research agenda","volume":"147","author":"Radianti","year":"2020","journal-title":"Comput. Educ."},{"key":"ref_6","unstructured":"iMotions (2025, November 27). Biometric Research Technology [Cheat Sheet]. Available online: https:\/\/imotions.com\/blog\/learning\/research-fundamentals\/biometric-research-technology\/."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Shu, L., Xie, J., Yang, M., Li, Z., Li, Z., Liao, D., Xu, X., and Yang, X. (2018). A review of emotion recognition using physiological signals. Sensors, 18.","DOI":"10.3390\/s18072074"},{"key":"ref_8","first-page":"2831","article-title":"Deep learning for emotion recognition: A survey","volume":"14","author":"Li","year":"2023","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_9","unstructured":"Stryker, C. (2025, November 24). What is Multimodal AI? IBM: Armonk, NY, USA, 2022. Available online: https:\/\/www.ibm.com\/think\/topics\/multimodal-ai."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"21923","DOI":"10.1007\/s00521-024-10371-3","article-title":"A review of multimodal-based emotion recognition techniques for cyberbullying detection in online social media platforms","volume":"36","author":"Wang","year":"2024","journal-title":"Neural Comput. Appl."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1109\/MSP.2017.2738401","article-title":"Deep multimodal learning: A survey on recent advances and trends","volume":"34","author":"Ramachandram","year":"2017","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Gadzicki, K., Khamsehashari, R., and Zetzsche, C. (2020, January 6\u20139). Early vs late fusion in multimodal convolutional neural networks. Proceedings of the 2020 23rd International Conference on Information Fusion (FUSION), Rustenburg, South Africa.","DOI":"10.23919\/FUSION45008.2020.9190246"},{"key":"ref_13","first-page":"22","article-title":"An\u00e1lisis comparativo de se\u00f1ales de ECG bajo problemas de salud y estados emocionales","volume":"10","year":"2022","journal-title":"P\u00e4di Bolet\u00edn Cient\u00edfico Cienc. B\u00e1sicas Ing. ICBI"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Nikolova, D., Petkova, P., Manolova, A., and Georgieva, P. (2018, January 25\u201326). ECG-based Emotion Recognition: Overview of Methods and Applications. Proceedings of the 2018 26th National Conference with International Participation (TELECOM), Varna, Bulgaria.","DOI":"10.1007\/978-3-030-23976-3_3"},{"key":"ref_15","first-page":"94","article-title":"La computaci\u00f3n afectiva: Emociones, tecnolog\u00edas y su relaci\u00f3n con la educaci\u00f3n virtual","volume":"5","author":"Bosquez","year":"2018","journal-title":"Rev. Investig. Talent."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/978-3-031-73347-5_1","article-title":"Upper-Body Hierarchical Graph for Skeleton Based Emotion Recognition in Assistive Driving","volume":"Volume 15084","author":"Wu","year":"2025","journal-title":"Proceedings of the Lecture Notes in Computer Science"},{"key":"ref_17","unstructured":"Aukstakalnis, S., Roth, S.F., and Blatner, D. (1993). El Espejismo de Silicio: Arte y Ciencia de la Realidad Virtual, Prentice Hall."},{"key":"ref_18","unstructured":"Universitat Oberta de Catalunya (UOC) (2019). Realidad Virtual, UOC Educational Repository. Available online: http:\/\/openaccess.uoc.edu."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Bastida, L., Sillaurren, S., Loizaga, E., Tom\u00e9, E., and Moya, A. (2024). Exploring Human Emotions: A Virtual Reality-Based Experimental Approach Integrating Physiological and Facial Analysis. Multimodal Technol. Interact., 8.","DOI":"10.3390\/mti8060047"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Saffaryazdi, N., Wasim, S.T., Dileep, K., Nia, A.F., Nanayakkara, S., Broadbent, E., and Billinghurst, M. (2022). Using Facial Micro-Expressions in Combination with EEG and Physiological Signals for Emotion Recognition. Front. Psychol., 13.","DOI":"10.3389\/fpsyg.2022.864047"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"106605","DOI":"10.1109\/ACCESS.2024.3436556","article-title":"Deep Representation Learning for Multimodal Emotion Recognition Using Physiological Signals","volume":"12","author":"Zubair","year":"2024","journal-title":"IEEE Access"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"2451","DOI":"10.1162\/089976600300015015","article-title":"Learning to forget: Continual prediction with LSTM","volume":"12","author":"Gers","year":"2000","journal-title":"Neural Comput."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Carri\u00f3n-Toro, M., Morales-Mart\u00ednez, D., Sant\u00f3rum, M., Bernal, L., Galindo, J., Aguilar, J., and Acosta-Vargas, P. (2025). Rethinking Usability in Serious Games: Designing an Instrument That Evaluates What Really Matters in Learning Contexts. Appl. Sci., 15.","DOI":"10.3390\/app151810084"},{"key":"ref_25","unstructured":"Google LLC (2025, November 15). MediaPipe Solutions: Multimodal Machine Learning Pipelines. Available online: https:\/\/github.com\/google-ai-edge\/mediapipe."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"6054","DOI":"10.1109\/JBHI.2024.3430310","article-title":"End-to-End Multimodal Emotion Recognition Based on Facial Expressions and Remote Photoplethysmography (rPPG) Signals","volume":"28","author":"Li","year":"2024","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s10055-024-00989-y","article-title":"Multimodal Emotion Classification Using Machine Learning in Immersive and Non-Immersive Virtual Reality","volume":"28","author":"Lima","year":"2024","journal-title":"Virtual Real."},{"key":"ref_28","unstructured":"Ahmad, Z., Rabbani, S., Zafar, R., Ishaque, S., Krishnan, S., and Khan, N. (2025). Multi-level Stress Assessment from ECG in a Virtual Reality Environment Using Multimodal Fusion. Technical Report, Escuela Polit\u00e9cnica Nacional."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Wang, L., Hao, J., and Zhou, T.H. (2023). ECG Multi-Emotion Recognition Based on Heart Rate Variability Signal Features Mining. Sensors, 23.","DOI":"10.3390\/s23208636"},{"key":"ref_30","unstructured":"Carri\u00f3n, M., Santorum, M., Aguilar, J., and Per\u00e9z, M. (2019). IPlus methodology for requirements elicitation for serious games. Ibero-American Conference on Software Engineering (CIbSE), Curran Associates."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1429","DOI":"10.1109\/TAI.2024.3523250","article-title":"A Hierarchical Cross-Modal Spatial Fusion Network for Multimodal Emotion Recognition","volume":"6","author":"Xu","year":"2025","journal-title":"IEEE Trans. Artif. Intell."},{"key":"ref_32","unstructured":"Goodfellow, I., Bengio, Y., and Courville, A. (2017). Deep Learning, The MIT Press."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1109\/TAFFC.2017.2764896","article-title":"Affective Recognition in Dynamic and Interactive Virtual Environments","volume":"11","author":"Moghimi","year":"2020","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_34","unstructured":"Benavides, J.S. (2026). Multimodal JSBC UIO Postural-ECG. Mendeley Data, V1."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/18\/5726\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,9]],"date-time":"2026-09-09T14:05:13Z","timestamp":1788962713000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/18\/5726"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,9]]},"references-count":34,"journal-issue":{"issue":"18","published-online":{"date-parts":[[2026,9]]}},"alternative-id":["s26185726"],"URL":"https:\/\/doi.org\/10.3390\/s26185726","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,9]]}}}