{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T19:32:41Z","timestamp":1783625561397,"version":"3.55.0"},"reference-count":106,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2021,1,24]],"date-time":"2021-01-24T00:00:00Z","timestamp":1611446400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001807","name":"Funda\u00e7\u00e3o de Amparo \u00e0 Pesquisa do Estado de S\u00e3o Paulo","doi-asserted-by":"publisher","award":["2017\/02377-5; 2018\/25902-0; 2017\/01687-0; 2013\/07375-0"],"award-info":[{"award-number":["2017\/02377-5; 2018\/25902-0; 2017\/01687-0; 2013\/07375-0"]}],"id":[{"id":"10.13039\/501100001807","id-type":"DOI","asserted-by":"publisher"}]},{"name":"METRICS","award":["H2020-ICT-2019-2-#871252"],"award-info":[{"award-number":["H2020-ICT-2019-2-#871252"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Worldwide demographic projections point to a progressively older population. This fact has fostered research on Ambient Assisted Living, which includes developments on smart homes and social robots. To endow such environments with truly autonomous behaviours, algorithms must extract semantically meaningful information from whichever sensor data is available. Human activity recognition is one of the most active fields of research within this context. Proposed approaches vary according to the input modality and the environments considered. Different from others, this paper addresses the problem of recognising heterogeneous activities of daily living centred in home environments considering simultaneously data from videos, wearable IMUs and ambient sensors. For this, two contributions are presented. The first is the creation of the Heriot-Watt University\/University of Sao Paulo (HWU-USP) activities dataset, which was recorded at the Robotic Assisted Living Testbed at Heriot-Watt University. This dataset differs from other multimodal datasets due to the fact that it consists of daily living activities with either periodical patterns or long-term dependencies, which are captured in a very rich and heterogeneous sensing environment. In particular, this dataset combines data from a humanoid robot\u2019s RGBD (RGB + depth) camera, with inertial sensors from wearable devices, and ambient sensors from a smart home. The second contribution is the proposal of a Deep Learning (DL) framework, which provides multimodal activity recognition based on videos, inertial sensors and ambient sensors from the smart home, on their own or fused to each other. The classification DL framework has also validated on our dataset and on the University of Texas at Dallas Multimodal Human Activities Dataset (UTD-MHAD), a widely used benchmark for activity recognition based on videos and inertial sensors, providing a comparative analysis between the results on the two datasets considered. Results demonstrate that the introduction of data from ambient sensors expressively improved the accuracy results.<\/jats:p>","DOI":"10.3390\/s21030768","type":"journal-article","created":{"date-parts":[[2021,1,25]],"date-time":"2021-01-25T12:28:31Z","timestamp":1611577711000},"page":"768","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":77,"title":["Activity Recognition for Ambient Assisted Living with Videos, Inertial Units and Ambient Sensors"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5680-9085","authenticated-orcid":false,"given":"Caetano Mazzoni","family":"Ranieri","sequence":"first","affiliation":[{"name":"Institute of Mathematical and Computer Sciences, University of Sao Paulo, Sao Carlos, SP 13566-590, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5824-672X","authenticated-orcid":false,"given":"Scott","family":"MacLeod","sequence":"additional","affiliation":[{"name":"Edinburgh Centre for Robotics, Heriot-Watt University, Edinburgh, EH14 4AS, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9013-2100","authenticated-orcid":false,"given":"Mauro","family":"Dragone","sequence":"additional","affiliation":[{"name":"Edinburgh Centre for Robotics, Heriot-Watt University, Edinburgh, EH14 4AS, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3272-2521","authenticated-orcid":false,"given":"Patricia Amancio","family":"Vargas","sequence":"additional","affiliation":[{"name":"Edinburgh Centre for Robotics, Heriot-Watt University, Edinburgh, EH14 4AS, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9366-2780","authenticated-orcid":false,"given":"Roseli\u00a0Aparecida Francelin","family":"Romero\u00a0","sequence":"additional","affiliation":[{"name":"Institute of Mathematical and Computer Sciences, University of Sao Paulo, Sao Carlos, SP 13566-590, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,1,24]]},"reference":[{"key":"ref_1","unstructured":"(2020, December 01). World Population Prospects 2019\u2014Population Division\u2014United Nations. Available online: https:\/\/www.un.org\/development\/desa\/publications\/world-population-prospects-2019-highlights.html."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1007\/s12652-016-0374-3","article-title":"Exploring the ambient assisted living domain: A systematic review","volume":"8","author":"Calvaresi","year":"2017","journal-title":"J. Ambient. Intell. Humaniz. Comput."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Maskeliunas, R., Dama\u0161evicius, R., and Segal, S. (2019). A review of internet of things technologies for ambient assisted living environments. Future Internet, 11.","DOI":"10.3390\/fi11120259"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Amato, G., Bacciu, D., Chessa, S., Dragone, M., Gallicchio, C., Gennaro, C., Lozano, H., Micheli, A., Hare, G.M.P.O., and Renteria, A. (2016). A Benchmark Dataset for Human Activity Recognition and Ambient Assisted Living. International Symposium on Ambient Intelligence, Springer.","DOI":"10.1007\/978-3-319-40114-0_1"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"11735","DOI":"10.3390\/s140711735","article-title":"A depth video sensor-based life-logging human activity recognition system for elderly care in smart indoor environments","volume":"14","author":"Jalal","year":"2014","journal-title":"Sensors"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Domb, M. (2019). Smart home systems based on internet of things. Internet of Things (IoT) for Automated and Smart Applications, IntechOpen.","DOI":"10.5772\/intechopen.84894"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Hasenauer, R., Belviso, C., and Ehrenmueller, I. (2019, January 23\u201325). New efficiency: Introducing social assistive robots in social eldercare organizations. Proceedings of the 2019 IEEE International Symposium on Innovation and Entrepreneurship, TEMS-ISIE 2019, Hangzhou, China.","DOI":"10.1109\/TEMS-ISIE46312.2019.9074296"},{"key":"ref_8","unstructured":"Cheng, L., Leung, A., and Ozawa, S. (2018). Deep feature learning and visualization for EEG recording using autoencoders. Neural Information Processing. ICONIP 2018. Lecture Notes in Computer Science (LNCS, volume 11307), Springer."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Fernandes Junior, F.E., Yang, G., Do, H.M., and Sheng, W. (2016, January 21\u201324). Detection of Privacy-sensitive Situations for Social Robots in Smart Homes. Proceedings of the 2016 IEEE International Conference on Automation Science and Engineering (CASE), Fort Worth, TX, USA.","DOI":"10.1109\/COASE.2016.7743474"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"698","DOI":"10.1016\/j.procs.2019.08.100","article-title":"Human activity recognition: A survey","volume":"155","author":"Jobanputra","year":"2019","journal-title":"Procedia Comput. Sci."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"10873","DOI":"10.1016\/j.eswa.2012.03.005","article-title":"A review on vision techniques applied to Human Behaviour Analysis for Ambient-Assisted Living","volume":"39","author":"Chaaraoui","year":"2012","journal-title":"Expert Syst. Appl."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1016\/j.image.2018.09.003","article-title":"TS-LSTM and temporal-inception: Exploiting spatiotemporal dynamics for activity recognition","volume":"71","author":"Ma","year":"2019","journal-title":"Signal Process. Image Commun."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Ahmed, A., Jalal, A., and Kim, K. (2020, January 14\u201318). RGB-D images for object segmentation, localization and recognition in indoor scenes using feature descriptor and Hough voting. Proceedings of the 2020 17th International Bhurban Conference on Applied Sciences and Technology (IBCAST), Islamabad, Pakistan.","DOI":"10.1109\/IBCAST47879.2020.9044545"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Sousa Lima, W., Souto, E., El-Khatib, K., Jalali, R., and Gama, J. (2019). Human Activity Recognition Using Inertial Sensors in a Smartphone: An Overview. Sensors, 19.","DOI":"10.3390\/s19143213"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Guo, J., Li, Y., Hou, M., Han, S., and Ren, J. (2020). Recognition of Daily Activities of Two Residents in a Smart Home Based on Time Clustering. Sensors, 20.","DOI":"10.3390\/s20051457"},{"key":"ref_16","unstructured":"Soomro, K., Zamir, A.R., and Shah, M. (2012). UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild. arXiv."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"754","DOI":"10.1016\/j.neucom.2015.07.085","article-title":"Transition-Aware Human Activity Recognition Using Smartphones","volume":"171","author":"Oneto","year":"2016","journal-title":"Neurocomputing"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., and Van Gool, L. (2016). Temporal segment networks: Towards good practices for deep action recognition. European Conference on Computer Vision (ECCV), Springer.","DOI":"10.1007\/978-3-319-46484-8_2"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Wang, P., Li, W., Gao, Z., Zhang, Y., Tang, C., and Ogunbona, P. (2017, January 21\u201326). Scene flow to action map: A new representation for RGB-D based action recognition with convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.52"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"99152","DOI":"10.1109\/ACCESS.2019.2927134","article-title":"A Hybrid Deep Learning Model for Human Activity Recognition Using Multimodal Body Sensing Data","volume":"7","author":"Gumaei","year":"2019","journal-title":"IEEE Access"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Du, Y., Lim, Y., and Tan, Y. (2019). A Novel Human Activity Recognition and Prediction in Smart Home Based on Interaction. Sensors, 19.","DOI":"10.3390\/s19204474"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1060","DOI":"10.1111\/coin.12233","article-title":"An ambient intelligence approach for learning in smart robotic environments","volume":"35","author":"Bacciu","year":"2019","journal-title":"Comput. Intell."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1016\/j.imavis.2017.01.010","article-title":"Going deeper into action recognition: A survey","volume":"60","author":"Herath","year":"2017","journal-title":"Image Vis. Comput."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Guesgen, H.W. (2020). Using Rough Sets to Improve Activity Recognition Based on Sensor Data. Sensors, 20.","DOI":"10.3390\/s20061779"},{"key":"ref_25","unstructured":"Ud din Tahir, S.B., Jalal, A., and Batool, M. (2020, January 17\u201319). Wearable Sensors for Activity Analysis using SMO-based Random Forest over Smart home and Sports Datasets. Proceedings of the 2020 3rd International Conference on Advancements in Computational Sciences (ICACS), Lahore, Pakistan."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1109\/5.554205","article-title":"An introduction to multisensor data fusion","volume":"85","author":"Hall","year":"1997","journal-title":"Proc. IEEE"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"4405","DOI":"10.1007\/s11042-015-3177-1","article-title":"A survey of depth and inertial sensor fusion for human action recognition","volume":"76","author":"Chen","year":"2017","journal-title":"Multimed. Tools Appl."},{"key":"ref_28","first-page":"568","article-title":"Two-stream convolutional networks for action recognition in videos","volume":"27","author":"Simonyan","year":"2014","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Farneb\u00e4ck, G. (2003). Two-Frame Motion Estimation Based on Polynomial Expansion. Scandinavian Conference on Image Analysis (SCIA), Springer.","DOI":"10.1007\/3-540-45103-X_50"},{"key":"ref_30","first-page":"25","article-title":"High accuracy optical flow estimation based on a theory for warping","volume":"Volume 3024","author":"Brox","year":"2004","journal-title":"European Conference on Computer Vision"},{"key":"ref_31","unstructured":"Zach, C., Pock, T., and Bischof, H. (2007). A duality based approach for real-time TV-L 1 optical flow. Joint Pattern Recognition Symposium, Springer."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"677","DOI":"10.1109\/TPAMI.2016.2599174","article-title":"Long-Term Recurrent Convolutional Networks for Visual Recognition and Description","volume":"39","author":"Donahue","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Ord\u00f3\u00f1ez, F., and Roggen, D. (2016). Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition. Sensors, 16.","DOI":"10.3390\/s16010115"},{"key":"ref_34","unstructured":"Garcia, F.A., Ranieri, C.M., and Romero, R.A.F. (2019, January 22\u201326). Temporal approaches for human activity recognition using inertial sensors. Proceedings of the 2019 Latin American Robotics Symposium (LARS), 2019 Brazilian Symposium on Robotics (SBR) and 2019 Workshop on Robotics in Education (WRE), Rio Grande, Brazil."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Song, S., Chandrasekhar, V., Mandal, B., Li, L., Lim, J.H., Babu, G.S., San, P.P., and Cheung, N.M. (July, January 26). Multimodal multi-stream deep learning for egocentric activity recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Las Vegas, NV, USA.","DOI":"10.1109\/CVPRW.2016.54"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Ranieri, C.M., Vargas, P.A., and Romero, R.A.F. (2020, January 19\u201324). Uncovering Human Multimodal Activity Recognition with a Deep Learning Approach. Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK.","DOI":"10.1109\/IJCNN48605.2020.9207255"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Chen, C., Jafari, R., and Kehtarnavaz, N. (2015, January 27\u201330). Utd-mhad: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor. Proceedings of the 2015 IEEE International conference on image processing (ICIP), Quebec City, QC, Canada.","DOI":"10.1109\/ICIP.2015.7350781"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., and Serre, T. (2011, January 6\u201313). HMDB: A large video database for human motion recognition. Proceedings of the 2011 IEEE International Conference on Computer Vision (ICCV), Barcelona, Spain.","DOI":"10.1109\/ICCV.2011.6126543"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Jiang, Y.G., Ye, G., Chang, S.F., Ellis, D., and Loui, A.C. (2011, January 8\u201311). Consumer video understanding: A benchmark database and an evaluation of human and machine performance. Proceedings of the 1st ACM International Conference on Multimedia Retrieval\u2014ICMR \u201911, New York, NY, USA.","DOI":"10.1145\/1991996.1992025"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Marszalek, M., Laptev, I., and Schmid, C. (2009, January 22\u201324). Actions in context. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Miami, FL, USA.","DOI":"10.1109\/CVPRW.2009.5206557"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., and Li, F.-F. (2014, January 24\u201327). Large-scale Video Classification with Convolutional Neural Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.223"},{"key":"ref_42","unstructured":"Carreira, J., Noland, E., Hillier, C., and Zisserman, A. (2019). A Short Note on the Kinetics-700 Human Action Dataset. arXiv."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.cviu.2016.10.018","article-title":"The THUMOS challenge on action recognition for videos \u201cin the wild\u201d","volume":"155","author":"Idrees","year":"2017","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Caba Heilbron, F., Escorcia, V., Ghanem, B., and Carlos Niebles, J. (2015, January 7\u201312). ActivityNet: A Large-Scale Video Benchmark for Human Activity Understanding. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298698"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1109\/MMUL.2012.24","article-title":"Microsoft kinect sensor and its effect","volume":"19","author":"Zhang","year":"2012","journal-title":"IEEE Multimed."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"1045","DOI":"10.1109\/TPAMI.2017.2691321","article-title":"Deep multimodal feature analysis for action recognition in RGB+D videos","volume":"40","author":"Shahroudy","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_47","first-page":"50","article-title":"Discriminative orderlet mining for real-time recognition of human-object interaction","volume":"Volume 9007","author":"Yu","year":"2015","journal-title":"Asian Conference on Computer Vision"},{"key":"ref_48","unstructured":"Wang, J., Liu, Z., Wu, Y., and Yuan, J. (2012, January 16\u201324). Mining actionlet ensemble for action recognition with depth cameras. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Oreifej, O., and Liu, Z. (2013, January 23\u201328). HON4D: Histogram of Oriented 4D Normals for Activity Recognition from Depth Sequences. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.","DOI":"10.1109\/CVPR.2013.98"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Ni, B., Wang, G., and Moulin, P. (2011, January 6\u201313). RGBD-HuDaAct: A color-depth video database for human daily activity recognition. Proceedings of the 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain.","DOI":"10.1109\/ICCVW.2011.6130379"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Liu, J., Shahroudy, A., Perez, M.L., Wang, G., Duan, L.Y., and Kot Chichung, A. (2019). NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding. IEEE Trans. Pattern Anal. Mach. Intell.","DOI":"10.1109\/TPAMI.2019.2916873"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"2033","DOI":"10.1016\/j.patrec.2012.12.014","article-title":"The Opportunity challenge: A benchmark database for on-body sensor-based activity recognition","volume":"34","author":"Chavarriaga","year":"2013","journal-title":"Pattern Recognit. Lett."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Reiss, A., and Stricker, D. (2012, January 18\u201322). Introducing a new benchmarked dataset for activity monitoring. Proceedings of the 2012 16th International Symposium on Wearable Computers, Newcastle, UK.","DOI":"10.1109\/ISWC.2012.13"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Ba\u00f1os, O., Damas, M., Pomares, H., Rojas, I., T\u00f3th, M.A., and Amft, O. (2012, January 5\u20138). A benchmark dataset to evaluate sensor displacement in activity recognition. Proceedings of the 2012 ACM Conference on Ubiquitous Computing\u2014UbiComp \u201912, New York, NY, USA.","DOI":"10.1145\/2370216.2370437"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Zappi, P., Lombriser, C., Stiefmeier, T., Farella, E., Roggen, D., Benini, L., and Tr\u00f6ster, G. (2008). Activity Recognition from On-Body Sensors: Accuracy-Power Trade-Off by Dynamic Sensor Selection. European Conference on Wireless Sensor Networks, Springer.","DOI":"10.1007\/978-3-540-77690-1_2"},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"B\u00e4chlin, M., Roggen, D., Tr\u00f6ster, G., Plotnik, M., Inbar, N., Meidan, I., Herman, T., Brozgol, M., Shaviv, E., and Giladi, N. (2009, January 4\u20137). Potentials of enhanced context awareness in wearable assistants for Parkinson\u2019s disease patients with the freezing of gait syndrome. Proceedings of the 2009 International Symposium on Wearable Computers, Linz, Austria.","DOI":"10.1109\/ISWC.2009.14"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1007\/978-3-319-21671-3_9","article-title":"Activity and anomaly detection in smart home: A survey","volume":"Volume 16","author":"Bakar","year":"2016","journal-title":"Next Generation Sensors and Systems"},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1109\/MC.2012.328","article-title":"CASAS: A Smart Home in a Box","volume":"46","author":"Cook","year":"2013","journal-title":"Computer"},{"key":"ref_59","doi-asserted-by":"crossref","unstructured":"Lesani, F.S., Fotouhi Ghazvini, F., and Amirkhani, H. (2019). Smart home resident identification based on behavioral patterns using ambient sensors. Pers. Ubiquitous Comput., 1\u201312.","DOI":"10.1007\/s00779-019-01288-z"},{"key":"ref_60","unstructured":"De la Torre Frade, F., Hodgins, J.K., Bargteil, A.W., Artal, X.M., Macey, J.C., Castells, A.C.I., and Beltran, J. (2008). Guide to the Carnegie Mellon University Multimodal Activity (CMU-MMAC) Database, Carnegie Mellon University. Tech. Rep. CMU-RI-TR-08-22."},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Ofli, F., Chaudhry, R., Kurillo, G., Vidal, R., and Bajcsy, R. (2013, January 15\u201317). Berkeley MHAD: A comprehensive Multimodal Human Action Database. Proceedings of the 2013 IEEE Workshop on Applications of Computer Vision (WACV), Tampa, FL, USA.","DOI":"10.1109\/WACV.2013.6474999"},{"key":"ref_62","doi-asserted-by":"crossref","unstructured":"Wei, H., Chopada, P., and Kehtarnavaz, N. (2020). C-MHAD: Continuous Multimodal Human Action Dataset of Simultaneous Video and Inertial Sensing. Sensors, 20.","DOI":"10.3390\/s20102905"},{"key":"ref_63","doi-asserted-by":"crossref","unstructured":"Stein, S., and Mckenna, S.J. (2013, January 8\u201312). Combining embedded accelerometers with computer vision for recognizing food preparation activities. Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), Zurich, Switzerland.","DOI":"10.1145\/2493432.2493482"},{"key":"ref_64","first-page":"220","article-title":"Towards automatic skill evaluation: Detection and segmentation of robot-assisted surgical motions","volume":"11","author":"Lin","year":"2010","journal-title":"Taylor Fr."},{"key":"ref_65","doi-asserted-by":"crossref","unstructured":"Ruffieux, S., Lalanne, D., and Mugellini, E. (2013, January 9\u201313). ChAirGest\u2014A Challenge for Multimodal Mid-Air Gesture Recognition for Close HCI. Proceedings of the 15th ACM on International conference on multimodal interaction\u2014ICMI \u201913, New York, NY, USA.","DOI":"10.1145\/2522848.2532590"},{"key":"ref_66","doi-asserted-by":"crossref","unstructured":"Kepski, M., and Kwolek, B. (2012). Fall Detection on Embedded Platform Using Kinect and Wireless Accelerometer. Comput. Help. People Spec. Needs, 407\u2013414.","DOI":"10.1007\/978-3-642-31534-3_60"},{"key":"ref_67","unstructured":"Gasparrini, S., Cippitelli, E., Gambi, E., Spinsante, S., W\u00e5hsl\u00e9n, J., Orhan, I., and Lindh, T. (2015). Proposal and experimental evaluation of fall detection solution based on wearable and depth data fusion. International Conference on ICT Innovations, Springer."},{"key":"ref_68","doi-asserted-by":"crossref","first-page":"368","DOI":"10.1109\/THMS.2016.2641388","article-title":"From Activity Recognition to Intention Recognition for Assisted Living Within Smart Homes","volume":"47","author":"Rafferty","year":"2017","journal-title":"IEEE Trans. Hum. Mach. Syst."},{"key":"ref_69","doi-asserted-by":"crossref","first-page":"3052","DOI":"10.1109\/TIP.2019.2955561","article-title":"Tangent Fisher vector on matrix manifolds for action recognition","volume":"29","author":"Luo","year":"2019","journal-title":"IEEE Trans. Image Process."},{"key":"ref_70","first-page":"3599","article-title":"Video Representation via Fusion of Static and Motion Features Applied to Human Activity Recognition","volume":"13","author":"Arif","year":"2019","journal-title":"KSII Trans. Internet Inf. Syst."},{"key":"ref_71","doi-asserted-by":"crossref","unstructured":"Nadeem, A., Jalal, A., and Kim, K. (2020, January 17\u201319). Human Actions Tracking and Recognition Based on Body Parts Detection via Artificial Neural Network. Proceedings of the 2020 3rd International Conference on Advancements in Computational Sciences (ICACS), Lahore, Pakistan.","DOI":"10.1109\/ICACS47775.2020.9055951"},{"key":"ref_72","doi-asserted-by":"crossref","unstructured":"Zhang, H.B., Zhang, Y.X., Zhong, B., Lei, Q., Yang, L., Du, J.X., and Chen, D.S. (2019). A Comprehensive Survey of Vision-Based Human Action Recognition Methods. Sensors, 19.","DOI":"10.3390\/s19051005"},{"key":"ref_73","doi-asserted-by":"crossref","unstructured":"Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015, January 7\u201313). Learning spatiotemporal features with 3D convolutional networks. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.510"},{"key":"ref_74","first-page":"832","article-title":"Human Activity Recognition using Convolutional 3D Network","volume":"2","author":"Anjali","year":"2019","journal-title":"Int. J. Res. Eng. Sci. Manag."},{"key":"ref_75","doi-asserted-by":"crossref","unstructured":"Wei, H., Jafari, R., and Kehtarnavaz, N. (2019). Fusion of Video and Inertial Sensing for Deep Learning\u2013Based Human Action Recognition. Sensors, 19.","DOI":"10.3390\/s19173680"},{"key":"ref_76","unstructured":"Jalal, A., Kim, J.T., and Kim, T.S. (2012, January 27\u201328). Development of a life logging system via depth imaging-based human activity recognition for smart homes. Proceedings of the International Symposium on Sustainable Healthy Buildings, Seoul, Korea."},{"key":"ref_77","doi-asserted-by":"crossref","unstructured":"Jalal, A., Kamal, S., and Kim, D. (2015, January 25\u201327). Shape and motion features approach for activity tracking and recognition from kinect video camera. Proceedings of the 2015 IEEE 29th International Conference on Advanced Information Networking and Applications Workshops, Gwangju, Korea.","DOI":"10.1109\/WAINA.2015.38"},{"key":"ref_78","doi-asserted-by":"crossref","first-page":"1857","DOI":"10.5370\/JEET.2016.11.6.1857","article-title":"Depth images-based human detection, tracking and activity recognition using spatiotemporal features and modified HMM","volume":"11","author":"Kamal","year":"2016","journal-title":"J. Electr. Eng. Technol."},{"key":"ref_79","doi-asserted-by":"crossref","unstructured":"Jalal, A., Kamal, S., and Kim, D. (2015, January 28\u201330). Depth Silhouettes Context: A new robust feature for human tracking and activity recognition based on embedded HMMs. Proceedings of the 2015 12th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI), Goyang City, Korea.","DOI":"10.1109\/URAI.2015.7358957"},{"key":"ref_80","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1016\/j.patcog.2016.08.003","article-title":"Robust human activity recognition from depth video using spatiotemporal multi-fused features","volume":"61","author":"Jalal","year":"2017","journal-title":"Pattern Recognit."},{"key":"ref_81","doi-asserted-by":"crossref","first-page":"2567","DOI":"10.1007\/s42835-019-00278-8","article-title":"Vision-based Human Activity recognition system using depth silhouettes: A Smart home system for monitoring the residents","volume":"14","author":"Kim","year":"2019","journal-title":"J. Electr. Eng. Technol."},{"key":"ref_82","doi-asserted-by":"crossref","unstructured":"Farooq, A., Jalal, A., and Kamal, S. (2015). Dense RGB-D Map-Based Human Tracking and Activity Recognition using Skin Joints Features and Self-Organizing Map. KSII Trans. Internet Inf. Syst., 9.","DOI":"10.3837\/tiis.2015.05.017"},{"key":"ref_83","doi-asserted-by":"crossref","first-page":"293","DOI":"10.1016\/j.patrec.2020.01.010","article-title":"A multimodal approach for human activity recognition based on skeleton and RGB data","volume":"131","author":"Franco","year":"2020","journal-title":"Pattern Recognit. Lett."},{"key":"ref_84","doi-asserted-by":"crossref","first-page":"498","DOI":"10.1109\/THMS.2015.2504550","article-title":"Action Recognition from Depth Maps Using Deep Convolutional Neural Networks","volume":"46","author":"Wang","year":"2016","journal-title":"IEEE Trans. Hum. Mach. Syst."},{"key":"ref_85","doi-asserted-by":"crossref","unstructured":"Jaimez, M., Souiai, M., Gonzalez-Jimenez, J., and Cremers, D. (2015, January 26\u201330). A Primal-Dual Framework for Real-Time Dense RGB-D Scene Flow. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Seattle, WA, USA.","DOI":"10.1109\/ICRA.2015.7138986"},{"key":"ref_86","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/j.patrec.2018.02.010","article-title":"Deep learning for sensor-based activity recognition: A survey","volume":"119","author":"Wang","year":"2019","journal-title":"Pattern Recognit. Lett."},{"key":"ref_87","doi-asserted-by":"crossref","first-page":"190","DOI":"10.1016\/j.engappai.2018.04.002","article-title":"Robust Human Activity Recognition using smartwatches and smartphones","volume":"72","author":"Blunck","year":"2018","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_88","doi-asserted-by":"crossref","first-page":"6061","DOI":"10.1007\/s11042-019-08463-7","article-title":"Wearable sensors based human behavioral pattern recognition using statistical features and reweighted genetic algorithm","volume":"79","author":"Quaid","year":"2020","journal-title":"Multimed. Tools Appl."},{"key":"ref_89","doi-asserted-by":"crossref","first-page":"94","DOI":"10.1016\/j.neucom.2019.06.081","article-title":"An adaptive and on-line IMU-based locomotion activity classification method using a triplet Markov model","volume":"362","author":"Li","year":"2019","journal-title":"Neurocomputing"},{"key":"ref_90","doi-asserted-by":"crossref","unstructured":"Rueda, F.M., and Fink, G.A. (2018, January 20\u201324). Learning attribute representation for human activity recognition. Proceedings of the 2018 24th International Conference on Pattern Recognition (ICPR), Beijing, China.","DOI":"10.1109\/ICPR.2018.8545146"},{"key":"ref_91","doi-asserted-by":"crossref","first-page":"961","DOI":"10.1109\/TKDE.2011.51","article-title":"A Knowledge-Driven Approach to Activity Recognition in Smart Homes","volume":"24","author":"Chen","year":"2012","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_92","doi-asserted-by":"crossref","first-page":"501","DOI":"10.1016\/j.neucom.2018.10.104","article-title":"A Sequential Deep Learning Application for Recognising Human Activities in Smart Homes","volume":"396","author":"Liciotti","year":"2019","journal-title":"Neurocomputing"},{"key":"ref_93","doi-asserted-by":"crossref","first-page":"441","DOI":"10.1016\/j.eswa.2018.07.068","article-title":"Ensemble classifier of long short-term memory with fuzzy temporal windows on binary sensors for activity recognition","volume":"114","author":"Zhang","year":"2018","journal-title":"Expert Syst. Appl."},{"key":"ref_94","first-page":"693","article-title":"Unobtrusive Activity Recognition of Elderly People Living Alone Using Anonymous Binary Sensors and DCNN","volume":"23","author":"Gochoo","year":"2019","journal-title":"IEEE J. Biomed. Health Inf."},{"key":"ref_95","doi-asserted-by":"crossref","unstructured":"Feichtenhofer, C., Pinz, A., and Zisserman, A. (2016, January 27\u201330). Convolutional two-stream network fusion for video action recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.213"},{"key":"ref_96","doi-asserted-by":"crossref","unstructured":"Yang, X., Ramesh, P., Chitta, R., Madhvanath, S., Bernal, E.A., and Luo, J. (2017, January 21\u201326). Deep Multimodal Representation Learning from Temporal Data. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.538"},{"key":"ref_97","unstructured":"(2020, July 23). Robotic Assisted Living Testbed. Available online: https:\/\/ralt.hw.ac.uk\/."},{"key":"ref_98","unstructured":"(2020, December 01). PAL Robotics. TIAGo Handbook Version 1.7.1. Available online: www.pal-robotics.com."},{"key":"ref_99","unstructured":"(2020, December 01). Astra Series\u2014Orbbec. Available online: https:\/\/orbbec3d.com\/product-astra-pro\/."},{"key":"ref_100","unstructured":"(2020, December 01). MetaMotionR\u2014MbientLab. Available online: https:\/\/mbientlab.com\/metamotionr\/."},{"key":"ref_101","first-page":"165","article-title":"On the Integration of Adaptive and Interactive Robotic Smart Spaces","volume":"6","author":"Dragone","year":"2015","journal-title":"Paladyn J. Behav. Robot."},{"key":"ref_102","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 19\u201325). Rethinking the inception architecture for computer vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.308"},{"key":"ref_103","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","article-title":"ImageNet large scale visual recognition challenge","volume":"115","author":"Russakovsky","year":"2015","journal-title":"Int. J. Comput. Vis."},{"key":"ref_104","unstructured":"El Madany, N.E.D., He, Y., and Guan, L. (2016, January 25\u201328). Human action recognition via multiview discriminative analysis of canonical correlations. Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA."},{"key":"ref_105","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1007\/s12652-019-01239-9","article-title":"Evaluating fusion of RGB-D and inertial sensors for multimodal human action recognition","volume":"11","author":"Imran","year":"2020","journal-title":"J. Ambient. Intell. Humaniz. Comput."},{"key":"ref_106","doi-asserted-by":"crossref","first-page":"11403","DOI":"10.1109\/JSEN.2019.2934678","article-title":"Autonomous Human Activity Classification from Wearable Multi-Modal Sensors","volume":"19","author":"Lu","year":"2019","journal-title":"IEEE Sens. J."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/3\/768\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:14:39Z","timestamp":1760159679000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/3\/768"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,1,24]]},"references-count":106,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2021,2]]}},"alternative-id":["s21030768"],"URL":"https:\/\/doi.org\/10.3390\/s21030768","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,1,24]]}}}