{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,13]],"date-time":"2026-03-13T09:12:34Z","timestamp":1773393154333,"version":"3.50.1"},"reference-count":39,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2016,10,15]],"date-time":"2016-10-15T00:00:00Z","timestamp":1476489600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Nature Science Foundation of China","award":["61602430"],"award-info":[{"award-number":["61602430"]}]},{"name":"National Nature Science Foundation of China","award":["61672475"],"award-info":[{"award-number":["61672475"]}]},{"name":"National Nature Science Foundation of China","award":["61402428"],"award-info":[{"award-number":["61402428"]}]},{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"publisher","award":["201513016"],"award-info":[{"award-number":["201513016"]}],"id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Human activity recognition is important for healthcare and lifestyle evaluation. In this paper, a novel method for activity recognition by jointly considering motion sensor data recorded by wearable smart watches and image data captured by RGB-Depth (RGB-D) cameras is presented. A normalized cross correlation based mapping method is implemented to establish association between motion sensor data with corresponding image data from the same person in multi-person situations. Further, to improve the performance and accuracy of recognition, a hierarchical structure embedded with an automatic group selection method is proposed. Through this method, if the number of activities to be classified is changed, the structure will be changed correspondingly without interaction. Our comparative experiments against the single data source and single layer methods have shown that our method is more accurate and robust.<\/jats:p>","DOI":"10.3390\/s16101713","type":"journal-article","created":{"date-parts":[[2016,10,17]],"date-time":"2016-10-17T10:33:16Z","timestamp":1476700396000},"page":"1713","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Hierarchical Activity Recognition Using Smart Watches and RGB-Depth Cameras"],"prefix":"10.3390","volume":"16","author":[{"given":"Zhen","family":"Li","sequence":"first","affiliation":[{"name":"College of Information Science and Engineering, Ocean University of China, Qingdao 266100, China"}]},{"given":"Zhiqiang","family":"Wei","sequence":"additional","affiliation":[{"name":"College of Information Science and Engineering, Ocean University of China, Qingdao 266100, China"}]},{"given":"Lei","family":"Huang","sequence":"additional","affiliation":[{"name":"College of Information Science and Engineering, Ocean University of China, Qingdao 266100, China"}]},{"given":"Shugang","family":"Zhang","sequence":"additional","affiliation":[{"name":"College of Information Science and Engineering, Ocean University of China, Qingdao 266100, China"}]},{"given":"Jie","family":"Nie","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China"}]}],"member":"1968","published-online":{"date-parts":[[2016,10,15]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"232","DOI":"10.1016\/j.cviu.2006.07.006","article-title":"A general method for human activity recognition in video","volume":"104","author":"Robertson","year":"2006","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"2564","DOI":"10.1109\/TIP.2010.2052823","article-title":"Roy-Chowdhury Amit K Tracking and Activity Recognition through Consensus in Distributed Camera Networks","volume":"19","author":"Bi","year":"2010","journal-title":"IEEE Trans. Image Process."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"8750","DOI":"10.3390\/s130708750","article-title":"Multi-view human activity recognition in distributed camera sensor networks","volume":"13","author":"Mosabbeb","year":"2013","journal-title":"Sensors"},{"key":"ref_4","first-page":"1","article-title":"Rgb-d camera-based daily living activity recognition","volume":"2","author":"Zhang","year":"2012","journal-title":"J. Comput. Vis. Image Process."},{"key":"ref_5","first-page":"1147","article-title":"GBD-HuDaAct: A color-depth video database for human daily activity recognition","volume":"47","author":"Ni","year":"2013","journal-title":"Adv. Comput. Vis. Pattern Recognit."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Fotiadou, E., and Nikolaidis, N. (2014, January 27\u201330). A correspondence based method for activity recognition in human skeleton motion sequences. Proceedings of the 2014 IEEE International Conference on Image Processing (ICIP), Paris, Fracne.","DOI":"10.1109\/ICIP.2014.7025300"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Zhu, G., Zhang, L., Shen, P., and Song, J. (2016). An Online Continuous Human Action Recognition Algorithm Based on the Kinect Sensor. Sensors, 16.","DOI":"10.3390\/s16020161"},{"key":"ref_8","unstructured":"Huang, W., Li, M., Hu, W., and Song, G. (2013, January 23\u201325). Cost sensitive GPS-based activity recognition. Proceedings of the International Conference on Fuzzy Systems and Knowledge Discovery, Shenyang, China."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1260\/2040-2295.6.1.1","article-title":"An exploratory study on a chest-worn computer for evaluation of diet, physical activity and lifestyle","volume":"6","author":"Sun","year":"2015","journal-title":"J. Healthc. Eng."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Chernbumroong, S., Atkins, A.S., and Yu, H. (2011, January 8\u201311). Activity classification using a single wrist-worn accelerometer. Proceedings of the 2011 5th International Conference on Software, Knowledge Information, Industrial Management and Applications (SKIMA), Benevento, Italy.","DOI":"10.1109\/SKIMA.2011.6089975"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"22500","DOI":"10.3390\/s141222500","article-title":"Long-Term Activity Recognition from Wristwatch Accelerometer Data","volume":"14","author":"Brena","year":"2014","journal-title":"Sensors"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"964","DOI":"10.1249\/MSS.0b013e31827f0d9c","article-title":"Estimating activity and sedentary behavior from an accelerometer on the hip or wrist","volume":"45","author":"Rosenberger","year":"2013","journal-title":"Med. Sci. Sports Exerc."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1166","DOI":"10.1109\/TITB.2010.2051955","article-title":"A triaxial accelerometer-based physical-activity recognition via augmented-signal features and a hierarchical recognizer","volume":"14","author":"Khan","year":"2010","journal-title":"IEEE Trans. Inf. Technol. Biomed."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"2193","DOI":"10.1249\/MSS.0b013e31829736d6","article-title":"Activity recognition using a single accelerometer placed at the wrist or ankle","volume":"45","author":"Mannini","year":"2013","journal-title":"Med. Sci. Sports Exerc."},{"key":"ref_15","unstructured":"Gao, L., Bourke, A.K., and Nelson, J. (2012, January 23\u201324). A comparison of classifiers for activity recognition using multiple accelerometer-based sensors. Proceedings of the IEEE 11th International Conference on Cybernetic Intelligent Systems, Limerick, Ireland."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Zhu, C., and Sheng, W. (2009, January 12\u201317). Human daily activity recognition in robot-assisted living using multi-sensor fusion. Proceedings of the IEEE International Conference on Robotics and Automation, Kebe, Japan.","DOI":"10.1109\/ROBOT.2009.5152756"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Bao, L., and Intille, S.S. (2004, January 18\u201323). Activity recognition from user-annotated acceleration data. Proceedings of the Second International Conference on Pervasive Computing, Vienna, Austria.","DOI":"10.1007\/978-3-540-24646-6_1"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"5163","DOI":"10.3390\/s150305163","article-title":"Low Energy Physical Activity Recognition System on Smartphones","volume":"15","author":"Morillo","year":"2015","journal-title":"Sensors"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Inoue, S., and Hattori, Y. (2011, January 19\u201322). Toward high-level activity recognition from accelerometers on mobile phones. Proceedings of the 4th International Conference on Cyber, Physical and Social Computing, Dalian, China.","DOI":"10.1109\/iThings\/CPSCom.2011.98"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1313","DOI":"10.1016\/j.engappai.2012.05.002","article-title":"State of the art of smart homes","volume":"25","author":"Silva","year":"2012","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., and Blake, A. (2011, January 20\u201325). Real-time human pose recognition in parts from single depth images. Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, CO, USA.","DOI":"10.1109\/CVPR.2011.5995316"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1007\/s11036-013-0448-9","article-title":"Energy-efficient motion related activity recognition on mobile devices for pervasive healthcare","volume":"19","author":"Liang","year":"2014","journal-title":"Mob. Networks Appl."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Klaser, A., Marszalek, M., and Schmid, C. (2008, January 1\u20134). A spatio-temporal descriptor based on 3d-gradients. Proceedings of the British MachineVision Conference, Leeds, UK.","DOI":"10.5244\/C.22.99"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"553","DOI":"10.1109\/JSTSP.2012.2193556","article-title":"A Local 3-D Motion Descriptor for Multi-View Human Action Recognition from 4-D Spatio-Temporal Interest Points","volume":"6","author":"Holte","year":"2012","journal-title":"IEEE J. Sel. Top. Signal Process."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Kantorov, V., and Laptev, I. (2014, January 24\u201327). Efficient Feature Extraction, Encoding, and Classification for Action Recognition. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, MT, USA.","DOI":"10.1109\/CVPR.2014.332"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1194","DOI":"10.1109\/TCYB.2014.2347057","article-title":"Multipe\/Single-View Human Action Recognition via Part-Induced Multitask Structural Learning","volume":"45","author":"Liu","year":"2015","journal-title":"IEEE Trans. Cybern."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1383","DOI":"10.1109\/TCYB.2013.2276433","article-title":"Multilevel depth and image fusion for human activity detection","volume":"43","author":"Ni","year":"2013","journal-title":"IEEE Trans. Cybern."},{"key":"ref_28","unstructured":"Wang, J., Liu, Z., Wu, Y., and Yuan, J. (2012, January 16\u201321). Mining actionlet ensemble for action recognition with depth cameras. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, RI, USA."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"252","DOI":"10.1155\/2011\/647858","article-title":"Accelerometry-based classification of human activities using Markov modeling","volume":"2011","author":"Mannini","year":"2011","journal-title":"Comput. Intell. Neurosci."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1109\/MSP.2006.1598086","article-title":"Pictures are not taken in a vacuum\u2014An overview of exploiting context for semantic scene content understanding","volume":"23","author":"Boutell","year":"2006","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1016\/j.cviu.2004.02.004","article-title":"Layered representations for learning and inferring office activity from multiple sensory channels","volume":"96","author":"Oliver","year":"2004","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1082","DOI":"10.1109\/TKDE.2007.1042","article-title":"Sensor-based abnormal human-activity detection","volume":"20","author":"Yin","year":"2008","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Shimosaka, M., Mori, T., and Sato, T. (2008, January 8\u201311). Robust indoor activity recognition via boosting. Proceedings of the 2008 19th International Conference on Pattern Recognition, Tampa, FL, USA.","DOI":"10.1109\/ICPR.2008.4761086"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Zeng, M., Nguyen, L.T., Yu, B., Mengshoel, O.J., Zhu, J., Wu, P., and Zhang, J. (2014, January 6\u20137). Convolutional Neural Networks for Human Activity Recognition Using Mobile Sensors. Proceedings of the 2014 6th International Conference on Mobile Computing, Applications and Services (MobiCASE), Austin, TX, USA.","DOI":"10.4108\/icst.mobicase.2014.257786"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"208","DOI":"10.1109\/TMM.2008.2009693","article-title":"Image Annotation Within the Context of Personal Photo Collections Using Hierarchical Event and Scene Models","volume":"11","author":"Cao","year":"2009","journal-title":"IEEE Trans. Multimed."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Yin, J., and Meng, Y. (2010, January 13\u201318). Human activity recognition in video using a hierarchical probabilistic latent model. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, San Francisco, CA, USA.","DOI":"10.1109\/CVPRW.2010.5543271"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Xia, L., Chen, C.C., and Aggarwal, J.K. (2012, January 16\u201321). View invariant human action recognition using histograms of 3D joints. Proceedings of the 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Providence, RI, USA.","DOI":"10.1109\/CVPRW.2012.6239233"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1109\/TBC.2008.2007456","article-title":"MPEG-7 Descriptors Based Shot Detection and Adaptive Initial Quantization Parameter Estimation for the H.264\/AVC","volume":"55","author":"Yang","year":"2009","journal-title":"IEEE Trans. Broadcast."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s10916-015-0239-x","article-title":"An Adaptive Hidden Markov Model for Activity Recognition Based on a Wearable Multi-Sensor Device","volume":"39","author":"Li","year":"2015","journal-title":"J. Med. Syst."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/16\/10\/1713\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T19:33:06Z","timestamp":1760211186000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/16\/10\/1713"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,10,15]]},"references-count":39,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2016,10]]}},"alternative-id":["s16101713"],"URL":"https:\/\/doi.org\/10.3390\/s16101713","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2016,10,15]]}}}