{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,19]],"date-time":"2026-08-19T04:27:03Z","timestamp":1787113623144,"version":"build-2736575974"},"reference-count":22,"publisher":"MDPI AG","issue":"16","license":[{"start":{"date-parts":[[2026,8,13]],"date-time":"2026-08-13T00:00:00Z","timestamp":1786579200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100004543","name":"China Scholarship Council","doi-asserted-by":"publisher","award":["202308390066"],"award-info":[{"award-number":["202308390066"]}],"id":[{"id":"10.13039\/501100004543","id-type":"DOI","asserted-by":"publisher"}]},{"award":["202308390066"],"award-info":[{"award-number":["202308390066"]}],"id":[{"id":"https:\/\/ror.org\/04atp4p48","id-type":"ROR","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>This article presents the implementation and technical feasibility evaluation of the multimodal sensing and feature-level fusion layer of BLEXER v3, a broader serious-game ecosystem under development for upper-limb rehabilitation. The implemented framework integrates Kinect-based motion tracking, Polar H10 and Bangle.js physiological sensing, wearable accelerometer data, and facial affective cues within a middleware-based architecture. Heterogeneous sensor streams are locally preprocessed, temporally aligned, and transformed into a common quality-aware multimodal feature representation containing motion, heart-rate and heart-rate-variability-related descriptors, affective information, availability indicators, signal-quality metadata, and freshness descriptors. The fused representation is additionally mapped, using predefined rules, to heuristic operational descriptors, including low demand, moderate stable, active engagement, physical load, affective activation, high demand, and uncertain. These descriptors are not intended as clinical diagnoses, independently validated user states, or final adaptation decisions. An exploratory K-means analysis of 12,675 complete multimodal windows reveals partial correspondence between the data-driven cluster structure and the predefined operational descriptors. Some descriptors show comparatively concentrated cluster patterns, whereas others exhibit overlap and internal heterogeneity. The results demonstrate the technical feasibility of generating structured multimodal representations that can provide input for subsequent context-aware reasoning. Independent validation of the operational descriptors, completion and evaluation of the whole system, and clinical validation with rehabilitation patients remain future work.<\/jats:p>","DOI":"10.3390\/s26165132","type":"journal-article","created":{"date-parts":[[2026,8,13]],"date-time":"2026-08-13T17:01:35Z","timestamp":1786640495000},"page":"5132","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Multimodal Data Fusion for a Self-Adaptive, Smart, Serious-Game Ecosystem Under Development"],"prefix":"10.3390","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-7872-2553","authenticated-orcid":false,"given":"Xiya","family":"Tao","sequence":"first","affiliation":[{"name":"Escuela de Ingenier\u00eda y Sistemas de Telecomunicaci\u00f3n (ETSIST), Centro de Electr\u00f3nica Industrial y Sistemas Multimodales (CEIMM), Universidad Polit\u00e9cnica de Madrid (UPM), Campus Sur, Calle Alan Turing s\/n, 28031 Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-9061-8685","authenticated-orcid":false,"given":"Peng","family":"Chen","sequence":"additional","affiliation":[{"name":"Escuela de Ingenier\u00eda y Sistemas de Telecomunicaci\u00f3n (ETSIST), Centro de Electr\u00f3nica Industrial y Sistemas Multimodales (CEIMM), Universidad Polit\u00e9cnica de Madrid (UPM), Campus Sur, Calle Alan Turing s\/n, 28031 Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4531-9918","authenticated-orcid":false,"given":"Martina","family":"Eckert","sequence":"additional","affiliation":[{"name":"Escuela de Ingenier\u00eda y Sistemas de Telecomunicaci\u00f3n (ETSIST), Centro de Electr\u00f3nica Industrial y Sistemas Multimodales (CEIMM), Universidad Polit\u00e9cnica de Madrid (UPM), Campus Sur, Calle Alan Turing s\/n, 28031 Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,8,13]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Eckert, M., Aglio, A., Mart\u2019in-Ruiz, M.L., and Osma-Ruiz, V. (2022). A New Architecture for Customizable Exergames: User Evaluation for Different Neuromuscular Disorders. Healthcare, 10.","DOI":"10.20944\/preprints202208.0184.v1"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"e36110","DOI":"10.2196\/36110","article-title":"Finding Effective Adjustment Levels for Upper Limb Exergames: Focus Group Study With Children With Physical Disabilities","volume":"11","author":"Eckert","year":"2023","journal-title":"JMIR Serious Games"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Tao, X., Chen, P., Tsami, M.E., Khayati, F., and Eckert, M. (2025, January 16\u201318). Design Process of a Self Adaptive Smart Serious Games Ecosystem. Proceedings of the IEEE International Conference on Intelligent Reality, Boston, MA, USA.","DOI":"10.1109\/ICIR68135.2025.11361599"},{"key":"ref_4","first-page":"55085","article-title":"The Blexer System\u2014Adaptive Full Play Therapeutic Exergames with Web-Based Supervision for People with Motor Dysfunctionalities","volume":"5","author":"Eckert","year":"2018","journal-title":"EAI Endorsed Trans. Serious Games"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Chen, P., Ceballos, S., Sancho, A.N., Moreno, S., Tao, X., and Eckert, M. (2025, January 16\u201318). Versatile VR Exercise Game Supporting AI Control\u2014Work in Progress. Proceedings of the IEEE International Conference on Intelligent Reality, Boston, MA, USA.","DOI":"10.1109\/ICIR68135.2025.11361610"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Schaffarczyk, M., Rogers, B., Reer, R., and Gronwald, T. (2022). Validity of the Polar H10 Sensor for Heart Rate Variability Analysis during Resting State and Incremental Exercise in Recreational Men and Women. Sensors, 22.","DOI":"10.3390\/s22176536"},{"key":"ref_7","unstructured":"Espruino (2026, August 10). Bangle.js Smartwatch. Available online: https:\/\/www.espruino.com\/Bangle.js."},{"key":"ref_8","unstructured":"Microsoft (2026, August 10). Azure Kinect DK Documentation. Available online: https:\/\/learn.microsoft.com\/en-us\/azure\/kinect-dk\/."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"T\u00f6lgyessy, M., Dekan, M., and Chovanec, L. (2021). Skeleton Tracking Accuracy and Precision Evaluation of Kinect V1, Kinect V2, and Azure Kinect. Appl. Sci., 11.","DOI":"10.3390\/app11125756"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., and Berg, A.C. (2016). SSD: Single Shot MultiBox Detector. Proceedings of the European Conference on Computer Vision, Springer.","DOI":"10.1007\/978-3-319-46448-0_2"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Goodfellow, I.J., Erhan, D., Carrier, P.L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., and Lee, D.H. (2013). Challenges in Representation Learning: A Report on Three Machine Learning Contests. Proceedings of the Neural Information Processing, Springer.","DOI":"10.1007\/978-3-642-42051-1_16"},{"key":"ref_12","unstructured":"Sambare, M. (2026, August 10). Facial Expression Recognition (FER-2013) Dataset. Available online: https:\/\/www.kaggle.com\/datasets\/msambare\/fer2013."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1161","DOI":"10.1037\/h0077714","article-title":"A Circumplex Model of Affect","volume":"39","author":"Russell","year":"1980","journal-title":"J. Personal. Soc. Psychol."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Song, X., van de Ven, S.S., Chen, S., Kang, P., Gao, Q., Jia, J., and Shull, P.B. (2022). Proposal of a Wearable Multimodal Sensing-Based Serious Games Approach for Hand Movement Training after Stroke. Front. Physiol., 13.","DOI":"10.3389\/fphys.2022.811950"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Chung, S., Lim, J., Noh, K.J., Kim, G., and Jeong, H. (2019). Sensor Data Acquisition and Multimodal Sensor Fusion for Human Activity Recognition Using Deep Learning. Sensors, 19.","DOI":"10.3390\/s19071716"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1016\/j.inffus.2023.01.015","article-title":"Multi-Level Feature Fusion for Multimodal Human Activity Recognition in Internet of Healthcare Things","volume":"94","author":"Islam","year":"2023","journal-title":"Inf. Fusion"},{"key":"ref_17","unstructured":"Barnum, G., Talukder, S., and Yue, Y. (2020). On the Benefits of Early Fusion in Multimodal Representation Learning. arXiv."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Roitberg, A., Peng, K., Marinov, Z., Seibold, C., Schneider, D., and Stiefelhagen, R. (2022). A Comparative Analysis of Decision-Level Fusion for Multimodal Driver Behaviour Understanding. arXiv.","DOI":"10.1109\/IV51971.2022.9827426"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1109\/TPAMI.2018.2798607","article-title":"Multimodal Machine Learning: A Survey and Taxonomy","volume":"41","author":"Ahuja","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1109\/MSP.2017.2738401","article-title":"Deep multimodal learning: A survey on recent advances and trends","volume":"34","author":"Ramachandram","year":"2017","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1016\/j.inffus.2016.09.005","article-title":"Multi-sensor fusion in body sensor networks: State-of-the-art and research challenges","volume":"35","author":"Gravina","year":"2017","journal-title":"Inf. Fusion"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"100696","DOI":"10.1016\/j.chbr.2025.100696","article-title":"Mapping the landscape of Artificial intelligence for serious games in Health: An enhanced meta review","volume":"18","author":"Tao","year":"2025","journal-title":"Comput. Hum. Behav. Rep."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/16\/5132\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,19]],"date-time":"2026-08-19T04:12:16Z","timestamp":1787112736000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/16\/5132"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,13]]},"references-count":22,"journal-issue":{"issue":"16","published-online":{"date-parts":[[2026,8]]}},"alternative-id":["s26165132"],"URL":"https:\/\/doi.org\/10.3390\/s26165132","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,13]]}}}