{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,1]],"date-time":"2026-06-01T23:31:52Z","timestamp":1780356712611,"version":"3.54.1"},"reference-count":43,"publisher":"SAGE Publications","issue":"1","license":[{"start":{"date-parts":[[2018,8,3]],"date-time":"2018-08-03T00:00:00Z","timestamp":1533254400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Journal of Intelligent &amp; Fuzzy Systems"],"published-print":{"date-parts":[[2019,2,16]]},"abstract":"<jats:p>Human action recognition in naturalistic videos is an important task with a broad range of applications. Recently, the encoder-decoder framework based on attention mechanism has been applied to action recognition. Although such conventional methods reach state-of-the-art, they always face a bottleneck of distinguishing similar actions. To solve this problem, we propose a novel recurrent attention convolutional neural network (RACNN), which incorporates convolutional neural networks (CNNs), long short-term memory (LSTM) and attention mechanism. Inspired by the composition of the action, the pre-action and the result of action might be important parts of an action, we introduce bi-direction LSTM with hierarchical structure. Additionally, the separated spatial-temporal attention is employed into our method. Furthermore, we find that incorporating spatio-temporal features extracted from three-dimensional CNNs (3DCNNs) and RGB features can enhance the relationship mined in each frame. Our comprehensive experimental results on two benchmark datasets, i.e., HMDB51 and UCF101, verify the effectiveness of our proposed methods and show that our proposals can significantly outperform the current state-of-the-art methods.<\/jats:p>","DOI":"10.3233\/jifs-18209","type":"journal-article","created":{"date-parts":[[2018,8,5]],"date-time":"2018-08-05T06:40:20Z","timestamp":1533451220000},"page":"775-786","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":13,"title":["Bi-direction hierarchical LSTM with spatial-temporal attention for action recognition"],"prefix":"10.1177","volume":"36","author":[{"given":"Haodong","family":"Yang","sequence":"first","affiliation":[{"name":"Science and Technology on Information Systems Engineering Laboratory, National University of Defense Technology, Changsha, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jun","family":"Zhang","sequence":"additional","affiliation":[{"name":"Science and Technology on Information Systems Engineering Laboratory, National University of Defense Technology, Changsha, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shuohao","family":"Li","sequence":"additional","affiliation":[{"name":"Science and Technology on Information Systems Engineering Laboratory, National University of Defense Technology, Changsha, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tingjin","family":"Luo","sequence":"additional","affiliation":[{"name":"College of Science, National University of Defense Technology, Changsha, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2018,8,3]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.imavis.2017.01.010"},{"key":"e_1_3_2_3_2","article-title":"Action recognition using visual attention","author":"Sharma S.","year":"2016","unstructured":"SharmaS., KirosR. and SalakhutdinovR., Action recognition using visual attention, ICLR Workshop (2016).","journal-title":"ICLR Workshop"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2012.59"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.502"},{"key":"e_1_3_2_6_2","first-page":"4489","article-title":"Learning spatiotemporal features with 3d convolutional networks","author":"Tran D.","year":"2015","unstructured":"TranD., BourdevL.D., FergusR., TorresaniL. and PaluriM., Learning spatiotemporal features with 3d convolutional networks, IEEE International Conference on Computer Vision (ICCV), 2015, pp. 4489\u20134497.","journal-title":"IEEE International Conference on Computer Vision (ICCV)"},{"issue":"4","key":"e_1_3_2_7_2","first-page":"568","article-title":"Two-stream convolutional networks for action recognition in videos","volume":"1","author":"Simonyan K.","year":"2014","unstructured":"SimonyanK. and ZissermanA., Two-stream convolutional networks for action recognition in videos, Advances in Neural Information Processing Systems 1(4) (2014), 568\u2013576.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_8_2","first-page":"1933","article-title":"Convolutional two-stream network fusion for video action recognition","author":"Feichtenhofer C.","year":"2016","unstructured":"FeichtenhoferC., PinzA. and ZissermanA., Convolutional two-stream network fusion for video action recognition, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 1933\u20131941.","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)"},{"issue":"1","key":"e_1_3_2_9_2","first-page":"20","article-title":"Temporal segment networks: Towards good practices for deep action recognition","volume":"22","author":"Wang L.","year":"2016","unstructured":"WangL., XiongY., WangZ., QiaoY., LinD., TangX. and GoolL.V., Temporal segment networks: Towards good practices for deep action recognition, Acm Transactions on Information Systems 22(1) (2016), 20\u201336.","journal-title":"Acm Transactions on Information Systems"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"e_1_3_2_11_2","first-page":"273","article-title":"Hybrid speech recognition with deep bidirectional lstm","author":"Graves A.","year":"2014","unstructured":"GravesA., JaitlyN. and MohamedA.-R., Hybrid speech recognition with deep bidirectional lstm, in Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on IEEE, 2014, pp. 273\u2013278.","journal-title":"in Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on IEEE"},{"key":"e_1_3_2_12_2","first-page":"3104","article-title":"Sequence to sequence learning with neural networks","author":"Sutskever I.","year":"2014","unstructured":"SutskeverI., VinyalsO. and LeQ.V., Sequence to sequence learning with neural networks, in Advances in Neural Information Processing Systems, 2014, pp. 3104\u20133112.","journal-title":"in Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.512"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cviu.2017.10.011"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2015.2477044"},{"key":"e_1_3_2_16_2","first-page":"2204","article-title":"Recurrent models of visual attention","volume":"3","author":"Mnih V.","year":"2014","unstructured":"MnihV., HeessN., GravesA. and KavukcuogluK., Recurrent models of visual attention, Advances in Neural Information Processing Systems 3 (2014), 2204\u20132212.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_17_2","article-title":"Attend it again: Recurrent attention convolutional neural network for action recognition","author":"Yang H.","year":"2018","unstructured":"YangH., ZhangJ. and LiS., Attend it again: Recurrent attention convolutional neural network for action recognition, Applied Science (2018).","journal-title":"Applied Science"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2011.6126543"},{"key":"e_1_3_2_19_2","article-title":"Ucf101: A dataset of 101 human actions classes from videos in the wild","author":"Soomro K.","year":"2012","unstructured":"SoomroK., ZamirA.R. and ShahM., Ucf101: A dataset of 101 human actions classes from videos in the wild, Computer Science (2012).","journal-title":"Computer Science"},{"key":"e_1_3_2_20_2","first-page":"1725","article-title":"Large-scale video classification with convolutional neural networks","author":"Karpathy A.","year":"2014","unstructured":"KarpathyA., TodericiG., ShettyS., LeungT., SukthankarR. and LiF.F., Large-scale video classification with convolutional neural networks, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 1725\u20131732.","journal-title":"in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)"},{"key":"e_1_3_2_21_2","article-title":"Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?","author":"Hara K.","year":"2017","unstructured":"HaraK., KataokaH. and SatohY., Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? in IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2017.","journal-title":"in IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR)"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/ROMA.2014.7295871"},{"key":"e_1_3_2_23_2","article-title":"Lstm neural reordering feature for statistical machine translation","volume":"2","author":"Cui Y.","year":"2016","unstructured":"CuiY., WangS., LiJ. and WangY., Lstm neural reordering feature for statistical machine translation, Computer Science 2 (2016).","journal-title":"Computer Science"},{"key":"e_1_3_2_24_2","first-page":"12","volume-title":"Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics","author":"Wang S.","year":"2015","unstructured":"WangS. and JiangJ., Learning natural language inference with lstm, Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics, 2015, p. 12."},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298878"},{"key":"e_1_3_2_26_2","first-page":"4694","article-title":"Beyond short snippets: Deep networks for video classification","author":"Ng J.Y.","year":"2015","unstructured":"NgJ.Y., HausknechtM.J., VijayanarasimhanS., VinyalsO., MongaR. and TodericiG., Beyond short snippets: Deep networks for video classification, IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 4694\u20134702.","journal-title":"IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR)"},{"key":"e_1_3_2_27_2","unstructured":"SunL. JiaK. ChenK. Yan YeungD. ShiB. and SavareseS. Lattice long short-term memory for human action recognition arXiv preprint arXiv:1708.03958 08 2017."},{"issue":"99","key":"e_1_3_2_28_2","first-page":"1","article-title":"Action recognition in video sequences using deep bi-directional lstm with cnn features","author":"Ullah A.","year":"2017","unstructured":"UllahA., AhmadJ., MuhammadK., SajjadM. and BaikS.W., Action recognition in video sequences using deep bi-directional lstm with cnn features, IEEE Access PP(99) (2017), 1\u20131.","journal-title":"IEEE Access"},{"key":"e_1_3_2_29_2","first-page":"3551","article-title":"Action recognition with improved trajectories","author":"Wang H.","year":"2013","unstructured":"WangH. and SchmidC., Action recognition with improved trajectories, in IEEE International Conference on Computer Vision (ICCV) IEEE, 2013, pp. 3551\u20133558.","journal-title":"in IEEE International Conference on Computer Vision (ICCV) IEEE"},{"key":"e_1_3_2_30_2","unstructured":"BaradelF. WolfC. and MilleJ. Pose-conditioned spatiotemporal attention for human action recognition arXiv preprint arXiv:1703.10106 2017."},{"key":"e_1_3_2_31_2","first-page":"2048","article-title":"Show, attend and tell: Neural image caption generation with visual attention","author":"Xu K.","year":"2015","unstructured":"XuK., BaJ., KirosR., ChoK., CourvilleA., SalakhutdinovR., ZemelR. and BengioY., Show, attend and tell: Neural image caption generation with visual attention, Computer Science (2015), 2048\u20132057.","journal-title":"Computer Science"},{"key":"e_1_3_2_32_2","first-page":"304","article-title":"Object attention patches for text detection and recognition in scene images using sift","author":"Sriman B.","year":"2015","unstructured":"SrimanB. and SchomakerL., Object attention patches for text detection and recognition in scene images using sift, in ICPRAM (1), 2015, pp. 304\u2013311.","journal-title":"in ICPRAM (1)"},{"key":"e_1_3_2_33_2","first-page":"1","article-title":"Every moment counts: Dense detailed labeling of actions in complex videos","volume":"8","author":"Yeung S.","year":"2015","unstructured":"YeungS., RussakovskyO., NingJ., AndrilukaM., MoriG. and LiF.F., Every moment counts: Dense detailed labeling of actions in complex videos, International Journal of Computer Vision (IJCV) 8 (2015), 1\u201315.","journal-title":"International Journal of Computer Vision (IJCV)"},{"key":"e_1_3_2_34_2","first-page":"843","article-title":"Unsupervised learning of video representations using lstms","author":"Srivastava N.","year":"2015","unstructured":"SrivastavaN., MansimovE. and SalakhudinovR., Unsupervised learning of video representations using lstms, International Conference on Machine Learning, 2015, pp. 843\u2013852.","journal-title":"International Conference on Machine Learning"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_2_36_2","first-page":"6645","article-title":"Speech recognition with deep recurrent neural networks","author":"Graves A.","year":"2013","unstructured":"GravesA., MohamedA.R. and HintonG., Speech recognition with deep recurrent neural networks, in IEEE International Conference on Acoustics, Speech and Signal Processing, 2013, pp. 6645\u20136649.","journal-title":"in IEEE International Conference on Acoustics, Speech and Signal Processing"},{"key":"e_1_3_2_37_2","unstructured":"AbadiM. AgarwalA. BarhamP. BrevdoE. ChenZ. CitroC. CorradoG.S. DavisA. DeanJ. DevinM. et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems arXiv: Distributed Parallel and Cluster Computing 2016."},{"key":"e_1_3_2_38_2","first-page":"4597","article-title":"Human action recognition using factorized spatio-temporal convolutional networks","author":"Sun L.","year":"2015","unstructured":"SunL., JiaK., YeungD.Y. and ShiB.E., Human action recognition using factorized spatio-temporal convolutional networks, in IEEE International Conference on Computer Vision (ICCV), 2015, pp. 4597\u20134605.","journal-title":"in IEEE International Conference on Computer Vision (ICCV)"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7299059"},{"key":"e_1_3_2_40_2","first-page":"7445","article-title":"Spatiotemporal multiplier networks for video action recognition","author":"Feichtenhofer C.","year":"2017","unstructured":"FeichtenhoferC., PinzA. and WildesR.P., Spatiotemporal multiplier networks for video action recognition, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) IEEE, 2017, pp. 7445\u20137454.","journal-title":"in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) IEEE"},{"issue":"247","key":"e_1_3_2_41_2","first-page":"1","article-title":"Visual categorization with bags of keypoints","volume":"44","author":"Csurka G.","year":"2004","unstructured":"CsurkaG., DanceC.R., FanL., WillamowskiJ. and BrayC., Visual categorization with bags of keypoints, Workshop on Statistical Learning in Computer Vision Eccv 44(247) (2004), 1\u201322.","journal-title":"Workshop on Statistical Learning in Computer Vision Eccv"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7299176"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298993"},{"key":"e_1_3_2_44_2","first-page":"2718","volume-title":"in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) IEEE","author":"Zhang B.","year":"2016","unstructured":"ZhangB., WangL., WangZ., QiaoY. and WangH., Realtime action recognition with enhanced motion vector cnns, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) IEEE, 2016, pp. 2718\u20132726."}],"container-title":["Journal of Intelligent &amp; Fuzzy Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.3233\/JIFS-18209","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.3233\/JIFS-18209","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.3233\/JIFS-18209","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T09:41:59Z","timestamp":1777455719000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.3233\/JIFS-18209"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,8,3]]},"references-count":43,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2019,2,16]]}},"alternative-id":["10.3233\/JIFS-18209"],"URL":"https:\/\/doi.org\/10.3233\/jifs-18209","relation":{},"ISSN":["1064-1246","1875-8967"],"issn-type":[{"value":"1064-1246","type":"print"},{"value":"1875-8967","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,8,3]]}}}