{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T23:41:13Z","timestamp":1784245273729,"version":"3.55.0"},"reference-count":45,"publisher":"MDPI AG","issue":"24","license":[{"start":{"date-parts":[[2021,12,12]],"date-time":"2021-12-12T00:00:00Z","timestamp":1639267200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003621","name":"Ministry of Science ICT and Future Planning","doi-asserted-by":"publisher","award":["2020R1A2C3011697"],"award-info":[{"award-number":["2020R1A2C3011697"]}],"id":[{"id":"10.13039\/501100003621","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>In recent years, human action recognition has been studied by many computer vision researchers. Recent studies have attempted to use two-stream networks using appearance and motion features, but most of these approaches focused on clip-level video action recognition. In contrast to traditional methods which generally used entire images, we propose a new human instance-level video action recognition framework. In this framework, we represent the instance-level features using human boxes and keypoints, and our action region features are used as the inputs of the temporal action head network, which makes our framework more discriminative. We also propose novel temporal action head networks consisting of various modules, which reflect various temporal dynamics well. In the experiment, the proposed models achieve comparable performance with the state-of-the-art approaches on two challenging datasets. Furthermore, we evaluate the proposed features and networks to verify the effectiveness of them. Finally, we analyze the confusion matrix and visualize the recognized actions at human instance level when there are several people.<\/jats:p>","DOI":"10.3390\/s21248309","type":"journal-article","created":{"date-parts":[[2021,12,13]],"date-time":"2021-12-13T01:29:33Z","timestamp":1639358973000},"page":"8309","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["An Efficient Human Instance-Guided Framework for Video Action Recognition"],"prefix":"10.3390","volume":"21","author":[{"given":"Inwoong","family":"Lee","sequence":"first","affiliation":[{"name":"Department of Electrical and Electronic Engineering, Yonsei University, Seoul 03722, Korea"},{"name":"Clova AI Research, NAVER Corporation, Seongnam 13561, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Doyoung","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Electrical and Electronic Engineering, Yonsei University, Seoul 03722, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dongyoon","family":"Wee","sequence":"additional","affiliation":[{"name":"Clova AI Research, NAVER Corporation, Seongnam 13561, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9895-5347","authenticated-orcid":false,"given":"Sanghoon","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Electrical and Electronic Engineering, Yonsei University, Seoul 03722, Korea"},{"name":"Department of Radiology, College of Medicine, Yonsei University, Seoul 03722, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,12,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"12496","DOI":"10.1109\/ACCESS.2017.2723039","article-title":"Implementation of a virtual training simulator based on 360\u00b0 multi-view human action recognition","volume":"5","author":"Kwon","year":"2017","journal-title":"IEEE Access"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"977","DOI":"10.1109\/83.931092","article-title":"Foveated video compression with optimal rate control","volume":"10","author":"Lee","year":"2001","journal-title":"IEEE Trans. Image Process."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1109\/6046.985561","article-title":"Foveated video quality assessment","volume":"4","author":"Lee","year":"2002","journal-title":"IEEE Trans. Multimed."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Lee, K., Lee, I., and Lee, S. (2018, January 8\u201314). Propagating lstm: 3d pose estimation based on joint interdependency. Proceedings of the European Conference on Computer Vision, Munich, Germany.","DOI":"10.1007\/978-3-030-01234-2_8"},{"key":"ref_5","unstructured":"Simonyan, K., and Zisserman, A. (2014). Two-Stream Convolutional Networks for Action Recognition in Videos. arXiv."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"2542","DOI":"10.1109\/TIP.2016.2548242","article-title":"Cross-view action recognition via transferable dictionary learning","volume":"25","author":"Zheng","year":"2016","journal-title":"IEEE Trans. Image Process."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Feichtenhofer, C., Pinz, A., and Wildes, R.P. (2017, January 21\u201326). Spatiotemporal multiplier networks for video action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.787"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"2799","DOI":"10.1109\/TIP.2018.2890749","article-title":"Action-stage emphasized spatiotemporal VLAD for video action recognition","volume":"28","author":"Tu","year":"2019","journal-title":"IEEE Trans. Image Process."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"He, K., Gkioxari, G., Doll\u00e1r, P., and Girshick, R. (2017, January 22\u201329). Mask r-cnn. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.322"},{"key":"ref_10","unstructured":"Newell, A., Huang, Z., and Deng, J. (2016). Associative embedding: End-to-end learning for joint detection and grouping. arXiv."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"130","DOI":"10.1109\/MSP.2017.2736018","article-title":"Deep convolutional neural models for picture-quality prediction: Challenges and solutions to data-driven image quality assessment","volume":"34","author":"Kim","year":"2017","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1109\/TNNLS.2018.2829819","article-title":"Deep CNN-based blind image quality predictor","volume":"30","author":"Kim","year":"2018","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Peng, X., and Schmid, C. (2016, January 8\u201316). Multi-region two-stream R-CNN for action detection. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46493-0_45"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Singh, G., Saha, S., Sapienza, M., Torr, P.H., and Cuzzolin, F. (2017, January 22\u201329). Online real-time multiple spatiotemporal action localisation and prediction. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.393"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Hou, R., Chen, C., and Shah, M. (2017, January 22\u201329). Tube convolutional neural network (t-cnn) for action detection in videos. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.620"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"He, J., Deng, Z., Ibrahim, M.S., and Mori, G. (2018, January 12\u201315). Generic tubelet proposals for action localization. Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, USA.","DOI":"10.1109\/WACV.2018.00044"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Li, D., Qiu, Z., Dai, Q., Yao, T., and Mei, T. (2018, January 8\u201314). Recurrent tubelet proposal and recognition networks for action detection. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01231-1_19"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Girdhar, R., Carreira, J., Doersch, C., and Zisserman, A. (2019, January 15\u201320). Video action transformer network. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00033"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Wu, J., Kuang, Z., Wang, L., Zhang, W., and Wu, G. (2020, January 23\u201328). Context-aware rcnn: A baseline for action detection in videos. Proceedings of the European Conference on Computer Vision, Glasgow, UK.","DOI":"10.1007\/978-3-030-58595-2_27"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Pan, J., Chen, S., Shou, M.Z., Liu, Y., Shao, J., and Li, H. (2021, January 19\u201325). Actor-context-actor relation network for spatio-temporal action localization. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00053"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Wang, L., Qiao, Y., and Tang, X. (2015, January 7\u201312). Action recognition with trajectory-pooled deep-convolutional descriptors. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7299059"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Girshick, R. (2015, January 7\u201313). Fast r-cnn. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.169"},{"key":"ref_24","first-page":"91","article-title":"Faster r-cnn: Towards real-time object detection with region proposal networks","volume":"28","author":"Ren","year":"2015","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Carreira, J., and Zisserman, A. (2017, January 21\u201326). Quo vadis, action recognition? A new model and the kinetics dataset. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.502"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., and Paluri, M. (2018, January 18\u201323). A closer look at spatiotemporal convolutions for action recognition. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00675"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Feichtenhofer, C., Pinz, A., and Zisserman, A. (2016, January 27\u201330). Convolutional two-stream network fusion for video action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.213"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"690","DOI":"10.1007\/s10489-020-01823-z","article-title":"A combined multiple action recognition and summarization for surveillance video sequences","volume":"51","author":"Elharrouss","year":"2021","journal-title":"Appl. Intell."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Lee, I., Kim, D., Kang, S., and Lee, S. (2017, January 22\u201329). Ensemble deep learning for skeleton-based action recognition using temporal sliding lstm networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.115"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"415","DOI":"10.1109\/TMM.2020.2978637","article-title":"3-D Human Behavior Understanding Using Generalized TS-LSTM Networks","volume":"23","author":"Lee","year":"2020","journal-title":"IEEE Trans. Multimed."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Lin, T.Y., Doll\u00e1r, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. (2017, January 21\u201326). Feature pyramid networks for object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.106"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Shahroudy, A., Liu, J., Ng, T.T., and Wang, G. (2016, January 27\u201330). Ntu rgb+ d: A large scale dataset for 3d human activity analysis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.115"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Wang, J., Nie, X., Xia, Y., Wu, Y., and Zhu, S.C. (2014, January 23\u201328). Cross-view action modeling, learning and recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.339"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll\u00e1r, P., and Zitnick, C.L. (2014, January 6\u201312). Microsoft coco: Common objects in context. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Baradel, F., Wolf, C., Mille, J., and Taylor, G.W. (2018, January 18\u201323). Glimpse clouds: Human activity recognition from unstructured feature points. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00056"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Wang, H., Song, Z., Li, W., and Wang, P. (2020). A hybrid network for large-scale action recognition from rgb and depth modalities. Sensors, 20.","DOI":"10.3390\/s20113305"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Shi, L., Zhang, Y., Cheng, J., and Lu, H. (2019, January 15\u201320). Skeleton-based action recognition with directed graph neural networks. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00810"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"R\u0103dulescu, B.A., and Florea, A.M. (2021, January 26\u201328). Human Action Recognition Methods Based on CNNs for RGB Video Input. Proceedings of the 2021 23rd International Conference on Control Systems and Computer Science (CSCS), Bucharest, Romania.","DOI":"10.1109\/CSCS52396.2021.00026"},{"key":"ref_39","unstructured":"Zhang, H., Li, Y., Wang, P., Liu, Y., and Shen, C. (2018). RGB-D based action recognition with light-weight 3D convolutional networks. arXiv."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Baptista, R., Ghorbel, E., Papadopoulos, K., Demisse, G.G., Aouada, D., and Ottersten, B. (2019, January 12\u201317). View-invariant action recognition from rgb data via 3d pose estimation. Proceedings of the ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.","DOI":"10.1109\/ICASSP.2019.8682904"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1016\/j.patcog.2017.02.030","article-title":"Enhanced skeleton visualization for view invariant human action recognition","volume":"68","author":"Liu","year":"2017","journal-title":"Pattern Recognit."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Donahue, J., Anne Hendricks, L., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., and Darrell, T. (2015, January 7\u201312). Long-term recurrent convolutional networks for visual recognition and description. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298878"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Rahmani, H., and Mian, A. (2015, January 7\u201312). Learning a non-linear knowledge transfer model for cross-view action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298860"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Jang, J., Kim, D., Park, C., Jang, M., Lee, J., and Kim, J. (2020, January 25\u201329). Etri-activity3d: A largescale rgb-d dataset for robots to recognize daily activities of the elderly. Proceedings of the International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA.","DOI":"10.1109\/IROS45743.2020.9341160"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Kim, D., Lee, I., Kim, D., and Lee, S. (2021). Action Recognition Using Close-Up of Maximum Activation and ETRI-Activity3D LivingLab Dataset. Sensors, 21.","DOI":"10.3390\/s21206774"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/24\/8309\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:46:01Z","timestamp":1760168761000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/24\/8309"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,12]]},"references-count":45,"journal-issue":{"issue":"24","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["s21248309"],"URL":"https:\/\/doi.org\/10.3390\/s21248309","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,12,12]]}}}