{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,9]],"date-time":"2026-04-09T06:15:34Z","timestamp":1775715334121,"version":"3.50.1"},"reference-count":43,"publisher":"MDPI AG","issue":"9","license":[{"start":{"date-parts":[[2018,9,6]],"date-time":"2018-09-06T00:00:00Z","timestamp":1536192000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100010661","name":"Horizon 2020","doi-asserted-by":"publisher","award":["737422"],"award-info":[{"award-number":["737422"]}],"id":[{"id":"10.13039\/100010661","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Detection of human activities along with the associated context is of key importance for various application areas, including assisted living and well-being. To predict a user\u2019s context in the daily-life situation a system needs to learn from multimodal data that are often imbalanced, and noisy with missing values. The model is likely to encounter missing sensors in real-life conditions as well (such as a user not wearing a smartwatch) and it fails to infer the context if any of the modalities used for training are missing. In this paper, we propose a method based on an adversarial autoencoder for handling missing sensory features and synthesizing realistic samples. We empirically demonstrate the capability of our method in comparison with classical approaches for filling in missing values on a large-scale activity recognition dataset collected in-the-wild. We develop a fully-connected classification network by extending an encoder and systematically evaluate its multi-label classification performance when several modalities are missing. Furthermore, we show class-conditional artificial data generation and its visual and quantitative analysis on context classification task; representing a strong generative power of adversarial autoencoders.<\/jats:p>","DOI":"10.3390\/s18092967","type":"journal-article","created":{"date-parts":[[2018,9,6]],"date-time":"2018-09-06T10:38:38Z","timestamp":1536230318000},"page":"2967","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":20,"title":["Synthesizing and Reconstructing Missing Sensory Modalities in Behavioral Context Recognition"],"prefix":"10.3390","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1473-0322","authenticated-orcid":false,"given":"Aaqib","family":"Saeed","sequence":"first","affiliation":[{"name":"Department of Mathematics and Computer Science, Eindhoven University of Technology, Eindhoven, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tanir","family":"Ozcelebi","sequence":"additional","affiliation":[{"name":"Department of Mathematics and Computer Science, Eindhoven University of Technology, Eindhoven, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Johan","family":"Lukkien","sequence":"additional","affiliation":[{"name":"Department of Mathematics and Computer Science, Eindhoven University of Technology, Eindhoven, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2018,9,6]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"579","DOI":"10.1109\/JBHI.2012.2234129","article-title":"A survey on ambient-assisted living tools for older adults","volume":"17","author":"Rashidi","year":"2013","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_2","unstructured":"Nahum-Shani, I., Smith, S.N., Tewari, A., Witkiewitz, K., Collins, L.M., Spring, B., and Murphy, S. (2014). Just in Time Adaptive Interventions (JITAIs): An Organizing Framework for Ongoing Health Behavior Support, The Methodology Center. Methodology Center Technical Report."},{"key":"ref_3","unstructured":"Avci, A., Bosch, S., Marin-Perianu, M., Marin-Perianu, R., and Havinga, P. (2010, January 22\u201323). Activity recognition using inertial sensing for healthcare, wellbeing and sports applications: A survey. Proceedings of the 23th International Conference on Architecture of Computing Systems, Hannover, Germany."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Rabbi, M., Aung, M.H., Zhang, M., and Choudhury, T. (2015, January 7\u201311). MyBehavior: Automatic personalized health feedback from user behaviors and preferences using smartphones. Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, Osaka, Japan.","DOI":"10.1145\/2750858.2805840"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"336","DOI":"10.1038\/nature23018","article-title":"Large-scale physical activity data reveal worldwide activity inequality","volume":"547","author":"Althoff","year":"2017","journal-title":"Nature"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"370","DOI":"10.1061\/(ASCE)CP.1943-5487.0000097","article-title":"Accelerometer-based activity recognition in construction","volume":"25","author":"Joshua","year":"2010","journal-title":"J. Comput. Civ. Eng."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Dey, A.K., Wac, K., Ferreira, D., Tassini, K., Hong, J.H., and Ramos, J. (2011, January 17\u201321). Getting closer: An empirical investigation of the proximity of user to their smart phones. Proceedings of the 13th International Conference on Ubiquitous Computing, Beijing, China.","DOI":"10.1145\/2030112.2030135"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1109\/MPRV.2017.3971131","article-title":"Recognizing Detailed Human Context in the Wild from Smartphones and Smartwatches","volume":"16","author":"Vaizman","year":"2017","journal-title":"IEEE Pervas. Comput."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"402","DOI":"10.4097\/kjae.2013.64.5.402","article-title":"The prevention and handling of the missing data","volume":"64","author":"Kang","year":"2013","journal-title":"Korean J. Anesthesiol."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Gelman, A., and Hill, J. (2006). Missing-data imputation. Data Analysis Using Regression and Multilevel\/Hierarchical Models, Cambridge University Press. Analytical Methods for Social Research.","DOI":"10.1017\/CBO9780511790942"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1798","DOI":"10.1109\/TPAMI.2013.50","article-title":"Representation learning: A review and new perspectives","volume":"35","author":"Bengio","year":"2013","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_12","first-page":"1929","article-title":"Dropout: A simple way to prevent neural networks from overfitting","volume":"15","author":"Srivastava","year":"2014","journal-title":"J. Mach. Learn. Res."},{"key":"ref_13","first-page":"3371","article-title":"Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion","volume":"11","author":"Vincent","year":"2010","journal-title":"J. Mach. Learn. Res."},{"key":"ref_14","unstructured":"Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. Advances in Neural Information Processing Systems 27, Proceedings of the Annual Conference on Neural Information Processing Systems, Montreal, QC, Canada, 8\u201313 December 2014, NIPS."},{"key":"ref_15","unstructured":"Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I., and Frey, B. (arXiv, 2015). Adversarial autoencoders, arXiv."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"5687","DOI":"10.3390\/s140305687","article-title":"Multi-sensor fusion for enhanced contextual awareness of everyday activities with ubiquitous devices","volume":"14","author":"Guiry","year":"2014","journal-title":"Sensors"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Wang, A., Chen, G., Shang, C., Zhang, M., and Liu, L. (2016, January 3\u20135). Human activity recognition in a smart home environment with stacked denoising autoencoders. Proceedings of the International Conference on Web-Age Information Management, Nanchang, China.","DOI":"10.1007\/978-3-319-47121-1_3"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Li, Y., Shi, D., Ding, B., and Liu, D. (2014). Unsupervised feature learning for human activity recognition using smartphone sensors. Mining Intelligence and Knowledge Exploration, Springer.","DOI":"10.1007\/978-3-319-13817-6_11"},{"key":"ref_19","unstructured":"Pl\u00f6tz, T., Hammerla, N.Y., and Olivier, P. (2011, January 16\u201322). Feature learning for activity recognition in ubiquitous computing. Proceedings of the IJCAI Proceedings\u2014International Joint Conference on Artificial Intelligence, Barcelona, Spain."},{"key":"ref_20","unstructured":"Wang, J., Chen, Y., Hao, S., Peng, X., and Hu, L. (arXiv, 2017). Deep learning for sensor-based activity recognition: A survey, arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"2413","DOI":"10.1109\/TCYB.2014.2373393","article-title":"Multilayer Joint Gait-Pose Manifolds for Human Gait Motion Modeling","volume":"45","author":"Ding","year":"2015","journal-title":"IEEE Trans. Cybern."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1540","DOI":"10.1109\/TCSVT.2016.2527218","article-title":"Video-based human walking estimation using joint gait and pose manifolds","volume":"27","author":"Zhang","year":"2017","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"4405","DOI":"10.1007\/s11042-015-3177-1","article-title":"A survey of depth and inertial sensor fusion for human action recognition","volume":"76","author":"Chen","year":"2017","journal-title":"Multimedia Tools Appl."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"168","DOI":"10.1145\/3161192","article-title":"Context Recognition In-the-Wild: Unified Model for Multi-Modal Sensors and Multi-Label Classification","volume":"1","author":"Vaizman","year":"2018","journal-title":"Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."},{"key":"ref_25","unstructured":"Thompson, B.B., Marks, R., and El-Sharkawi, M.A. (2003, January 20\u201324). On the contractive nature of autoencoders: Application to missing sensor restoration. Proceedings of the International Joint Conference on Neural Networks, Portland, OR, USA."},{"key":"ref_26","unstructured":"Nelwamondo, F.V., Mohamed, S., and Marwala, T. (arXiv, 2007). Missing data: A comparison of neural network and expectation maximization techniques, arXiv."},{"key":"ref_27","unstructured":"Duan, Y., Lv, Y., Kang, W., and Zhao, Y. (2014, January 8\u201311). A deep learning based approach for traffic data imputation. Proceedings of the 17th International IEEE Conference on Intelligent Transportation Systems (ITSC), Qingdao, China."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Beaulieu-Jones, B.K., and Moore, J.H. (2017, January 3\u20137). Missing data imputation in the electronic health record using deeply learned autoencoders. Proceedings of the Pacific Symposium on Biocomputing 2017, Big Island of Hawaii, HI, USA.","DOI":"10.1142\/9789813207813_0021"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Jaques, N., Taylor, S., Sano, A., and Picard, R. (2017, January 23\u201326). Multimodal Autoencoder: A Deep Learning Approach to Filling in Missing Sensor Data and Enabling Better Mood Prediction. Proceedings of the International Conference on Affective Computing and Intelligent Interaction (ACII), San Antonio, TX, USA.","DOI":"10.1109\/ACII.2017.8273601"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1016\/j.neucom.2014.08.092","article-title":"Feature learning from incomplete EEG with denoising autoencoder","volume":"165","author":"Li","year":"2015","journal-title":"Neurocomputing"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"26094","DOI":"10.1038\/srep26094","article-title":"Deep patient: An unsupervised representation to predict the future of patients from the electronic health records","volume":"6","author":"Miotto","year":"2016","journal-title":"Sci. Rep."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1109\/MCI.2013.2247823","article-title":"Learning deep physiological models of affect","volume":"8","author":"Martinez","year":"2013","journal-title":"IEEE Comput. Intell. Mag."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"500","DOI":"10.1109\/LSP.2017.2672753","article-title":"Universum autoencoder-based domain adaptation for speech emotion recognition","volume":"24","author":"Deng","year":"2017","journal-title":"IEEE Signal Process. Lett."},{"key":"ref_34","unstructured":"Kuchaiev, O., and Ginsburg, B. (arXiv, 2017). Training Deep AutoEncoders for Collaborative Filtering, arXiv."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Yu, L., Zhang, W., Wang, J., and Yu, Y. (2017, January 4\u20139). SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient. Proceedings of the AAAI, San Francisco, CA, USA.","DOI":"10.1609\/aaai.v31i1.10804"},{"key":"ref_36","unstructured":"Choi, E., Biswal, S., Malin, B., Duke, J., Stewart, W.F., and Sun, J. (arXiv, 2017). Generating multi-label discrete electronic health records using generative adversarial networks, arXiv."},{"key":"ref_37","unstructured":"Esteban, C., Hyland, S.L., and R\u00e4tsch, G. (arXiv, 2017). Real-valued (medical) time series generation with recurrent conditional GANs, arXiv."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"504","DOI":"10.1126\/science.1127647","article-title":"Reducing the dimensionality of data with neural networks","volume":"313","author":"Hinton","year":"2006","journal-title":"Science"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Nam, J., Kim, J., Menc\u00eda, E.L., Gurevych, I., and F\u00fcrnkranz, J. (2014, January 15\u201319). Large-scale multi-label text classification\u2013revisiting neural networks. Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Nancy, France.","DOI":"10.1007\/978-3-662-44851-9_28"},{"key":"ref_40","first-page":"265","article-title":"TensorFlow: A System for Large-Scale Machine Learning","volume":"16","author":"Abadi","year":"2016","journal-title":"OSDI"},{"key":"ref_41","unstructured":"Glorot, X., and Bengio, Y. (2010, January 13\u201315). Understanding the difficulty of training deep feedforward neural networks. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Sardinia, Italy."},{"key":"ref_42","unstructured":"Kingma, D.P., and Ba, J. (arXiv, 2014). Adam: A method for stochastic optimization, arXiv."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Abadi, M., Chu, A., Goodfellow, I., McMahan, H.B., Mironov, I., Talwar, K., and Zhang, L. (2016, January 24\u201328). Deep learning with differential privacy. Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria.","DOI":"10.1145\/2976749.2978318"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/18\/9\/2967\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T15:19:05Z","timestamp":1760195945000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/18\/9\/2967"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,9,6]]},"references-count":43,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2018,9]]}},"alternative-id":["s18092967"],"URL":"https:\/\/doi.org\/10.3390\/s18092967","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,9,6]]}}}