{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T02:56:09Z","timestamp":1777431369959,"version":"3.51.4"},"reference-count":56,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2025,7,15]],"date-time":"2025-07-15T00:00:00Z","timestamp":1752537600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Comput. Sci."],"abstract":"<jats:sec><jats:title>Introduction<\/jats:title><jats:p>Human action recognition is a critical task with broad applications and remains a challenging problem due to the complexity of modeling dynamic interactions between individuals. Existing methods, including skeleton sequence-based and RGB video-based models, have achieved impressive accuracy but often suffer from high computational costs and limited effectiveness in modeling human interaction behaviors.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>To address these limitations, we propose a lightweight Convolutional Spatiotemporal Sequence Inference Model (CSSIModel) for recognizing human interaction behaviors. The model extracts features from skeleton sequences using DINet and from RGB video frames using ResNet-18. These multi-modal features are fused and processed using a novel multiscale two-dimensional convolutional peak-valley inference module to classify interaction behaviors.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>CSSIModel achieves competitive results across several benchmark datasets: 87.4% accuracy on NTU RGB+D 60 (XSub), 94.1% on NTU RGB+D 60 (XView), 80.5% on NTU RGB+D 120 (XSub), and 84.9% on NTU RGB+D 120 (XSet). These results are comparable to or exceed those of state-of-the-art methods.<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion<\/jats:title><jats:p>The proposed method effectively balances accuracy and computational efficiency. By significantly reducing model complexity while maintaining high performance, CSSIModel is well-suited for real-time applications and provides a valuable reference for future research in multi-modal human behavior recognition.<\/jats:p><\/jats:sec>","DOI":"10.3389\/fcomp.2025.1576775","type":"journal-article","created":{"date-parts":[[2025,7,15]],"date-time":"2025-07-15T05:24:56Z","timestamp":1752557096000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Convolutional spatio-temporal sequential inference model for human interaction behavior recognition"],"prefix":"10.3389","volume":"7","author":[{"given":"Lizhong","family":"Jin","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rulong","family":"Fan","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiaoling","family":"Han","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xueying","family":"Cui","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2025,7,15]]},"reference":[{"key":"ref1","author":"Arnab","year":"2021"},{"key":"ref2","author":"Bertasius","year":"2021"},{"key":"ref3","author":"Carreira","year":"2017"},{"key":"ref4","author":"Cheng","year":"2020"},{"key":"ref5","doi-asserted-by":"publisher","first-page":"168297","DOI":"10.1109\/ACCESS.2020.3023599","article-title":"Infrared and 3D skeleton feature fusion for RGB-D action recognition","volume":"8","author":"De Boissiere","year":"2020","journal-title":"IEEE Access"},{"key":"ref6","author":"Ding","year":"2019"},{"key":"ref7","author":"Duan","year":""},{"key":"ref8","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR52688.2022.00298","article-title":"Revisiting skeleton-based action recognition","author":"Duan","year":""},{"key":"ref9","author":"Gao","year":"2019"},{"key":"ref10","author":"He","year":"2016"},{"key":"ref11","doi-asserted-by":"publisher","first-page":"109528","DOI":"10.1016\/j.patcog.2023.109528","article-title":"Continual spatio-temporal graph convolutional networks","volume":"140","author":"Hedegaard","year":"2023","journal-title":"Pattern Recogn."},{"key":"ref12","author":"Ji","year":"2014"},{"key":"ref9501","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2212.05638","article-title":"\u201cCross-modal learning with 3D deformable attention for action recognition.\u201d","author":"Kim","year":"2022","journal-title":"arXiv"},{"key":"ref13","doi-asserted-by":"publisher","first-page":"198","DOI":"10.1016\/j.patcog.2018.08.006","article-title":"Prediction of partially observed human activity based on pre-trained deep representation","volume":"85","author":"Lee","year":"2019","journal-title":"Pattern Recogn."},{"key":"ref14","doi-asserted-by":"publisher","first-page":"108645","DOI":"10.1016\/j.patcog.2022.108645","article-title":"Human interaction recognition framework based on interacting body part attention","volume":"128","author":"Lee","year":"2022","journal-title":"Pattern Recogn."},{"key":"ref15","author":"Li","year":"2019"},{"key":"ref16","author":"Li","year":"2017"},{"key":"ref17","first-page":"12026","article-title":"Two-stream adaptive graph convolutional networks for skeleton-based action recognition","volume-title":"Proc. IEEE\/CVF conference on computer vision and pattern recognition (CVPR)","author":"Li","year":"2019"},{"key":"ref18","doi-asserted-by":"crossref","first-page":"109231","DOI":"10.1016\/j.patcog.2022.109231","article-title":"Spatiotemporal focus for skeleton-based action recognition","volume":"136","author":"Li","year":"2023","journal-title":"Pattern Recogn."},{"key":"ref19","author":"Lin","year":"2023"},{"key":"ref20","doi-asserted-by":"publisher","first-page":"287","DOI":"10.1016\/j.neucom.2018.08.066","article-title":"A structured multi-feature representation for recognizing human action and interaction","volume":"318","author":"Liu","year":"2018","journal-title":"Neurocomputing"},{"key":"ref21","doi-asserted-by":"publisher","first-page":"2684","DOI":"10.1109\/tpami.2019.2916873","article-title":"NTU RGB+D 120: a large-scale benchmark for 3D human activity understanding","volume":"42","author":"Liu","year":"","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref22","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1016\/j.neucom.2019.11.048","article-title":"HDS-SP: a novel descriptor for skeleton-based human action recognition","volume":"385","author":"Liu","year":"","journal-title":"Neurocomputing"},{"key":"ref23","author":"Pang","year":"2022"},{"key":"ref24","doi-asserted-by":"publisher","first-page":"645952","DOI":"10.3389\/fnhum.2021.645952","article-title":"Data augmentation: using channel-level recombination to improve classification performance for motor imagery EEG","volume":"15","author":"Pei","year":"","journal-title":"Front. Hum. Neurosci."},{"key":"ref25","doi-asserted-by":"crossref","first-page":"465","DOI":"10.1109\/TNSRE.2021.3125386","article-title":"A tensor-based frequency features combination method for brain-computer interfaces","volume":"30","author":"Pei","year":"","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref26","doi-asserted-by":"publisher","first-page":"026038","DOI":"10.1088\/1741-2552\/adbfc0","article-title":"Toward the enhancement of affective brain-computer interfaces using dependence within EEG series","volume":"22","author":"Pei","year":"","journal-title":"J. Neural Eng."},{"key":"ref27","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2025.3551330","article-title":"Identifying stable EEG patterns in manipulation task for negative emotion recognition","author":"Pei","year":"","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref28","doi-asserted-by":"crossref","first-page":"366","DOI":"10.1109\/TMM.2021.3050642","article-title":"Interaction relational network for mutual action recognition","volume":"24","author":"Perez","year":"2021","journal-title":"IEEE Trans. Multimed."},{"key":"ref29","doi-asserted-by":"publisher","first-page":"103219","DOI":"10.1016\/j.cviu.2021.103219","article-title":"Skeleton-based action recognition via spatial and temporal transformer networks","author":"Plizzari","year":"2021","journal-title":"Comput. Vis. Image Underst."},{"key":"ref30","author":"Santoro","year":"2017"},{"key":"ref31","author":"Shahroudy","year":"2016"},{"key":"ref32","author":"Shi","year":"2018"},{"key":"ref33","author":"Shi","year":"2019"},{"key":"ref34","author":"Shi","year":"2021"},{"key":"ref35","doi-asserted-by":"crossref","first-page":"1110","DOI":"10.1109\/TPAMI.2019.2942030","article-title":"Hierarchical long short-term concurrent memory for human interaction recognition","volume":"43","author":"Shu","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref36","author":"Shu","year":"2017"},{"key":"ref37","author":"Simonyan","year":"2014"},{"key":"ref38","author":"Simonyan","year":"2015"},{"key":"ref39","author":"Song","year":"2022"},{"key":"ref40","doi-asserted-by":"publisher","first-page":"1915","DOI":"10.1109\/TCSVT.2020.3015051","article-title":"Richly activated graph convolutional network for robust skeleton-based action recognition","volume":"31","author":"Song","year":"2021","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref41","doi-asserted-by":"publisher","first-page":"39505","DOI":"10.1109\/ACCESS.2024.3377103","article-title":"Auto GCN-toward generic human activity recognition with neural architecture search","volume":"12","author":"Tempel","year":"2024","journal-title":"IEEE Access"},{"key":"ref42","author":"Tran","year":"2015"},{"key":"ref43","author":"Tran","year":"2018"},{"key":"ref44","author":"Trivedi","year":"2022"},{"key":"ref45","doi-asserted-by":"publisher","first-page":"018","DOI":"10.31083\/j.jin2301018","article-title":"Fusion of multi-domain EEG signatures improves emotion recognition","volume":"23","author":"Wang","year":"2023","journal-title":"J. Integr. Neurosci."},{"key":"ref46","author":"Wu","year":"2018"},{"key":"ref47","doi-asserted-by":"publisher","first-page":"2866","DOI":"10.1609\/aaai.v36i3.20191","article-title":"Topology-aware convolutional neural network for efficient skeleton-based action recognition","volume":"36","author":"Xu","year":"2022","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"ref48","author":"Xu","year":"2020"},{"key":"ref49","doi-asserted-by":"publisher","first-page":"12328","DOI":"10.1609\/aaai.v32i1.12328","article-title":"Spatial temporal graph convolutional networks for skeleton-based action recognition","volume":"32","author":"Yan","year":"2018","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"ref50","doi-asserted-by":"publisher","first-page":"10040","DOI":"10.1109\/ACCESS.2020.2964115","article-title":"PGCN-TCA: pseudo graph convolutional network with temporal and channel-wise attention for skeleton-based action recognition","volume":"8","author":"Yang","year":"2020","journal-title":"IEEE Access"},{"key":"ref51","author":"Yun","year":"2012"},{"key":"ref52","author":"Zhang","year":"2017"},{"key":"ref53","author":"Zhang","year":"2018"},{"key":"ref54","author":"Zhang","year":"2022"},{"key":"ref55","author":"Zhou","year":"2018"}],"container-title":["Frontiers in Computer Science"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fcomp.2025.1576775\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,15]],"date-time":"2025-07-15T05:24:57Z","timestamp":1752557097000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fcomp.2025.1576775\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,15]]},"references-count":56,"alternative-id":["10.3389\/fcomp.2025.1576775"],"URL":"https:\/\/doi.org\/10.3389\/fcomp.2025.1576775","relation":{},"ISSN":["2624-9898"],"issn-type":[{"value":"2624-9898","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,15]]},"article-number":"1576775"}}