{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,10]],"date-time":"2026-01-10T01:56:23Z","timestamp":1768010183750,"version":"3.49.0"},"reference-count":89,"publisher":"World Scientific Pub Co Pte Ltd","issue":"01","funder":[{"name":"Regional Operational Programme Ionian Islands","award":["2014-2020"],"award-info":[{"award-number":["2014-2020"]}]},{"name":"TRaditional corfU Music PresErvation through digiTal innovation","award":["5030952"],"award-info":[{"award-number":["5030952"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Int. J. Neur. Syst."],"published-print":{"date-parts":[[2023,1]]},"abstract":"<jats:p> The problem of human activity recognition (HAR) has been increasingly attracting the efforts of the research community, having several applications. It consists of recognizing human motion and\/or behavior within a given image or a video sequence, using as input raw sensor measurements. In this paper, a multimodal approach addressing the task of video-based HAR is proposed. It is based on 3D visual data that are collected using an RGB + depth camera, resulting to both raw video and 3D skeletal sequences. These data are transformed into six different 2D image representations; four of them are in the spectral domain, another is a pseudo-colored image. The aforementioned representations are based on skeletal data. The last representation is a \u201cdynamic\u201d image which is actually an artificially created image that summarizes RGB data of the whole video sequence, in a visually comprehensible way. In order to classify a given activity video, first, all the aforementioned 2D images are extracted and then six trained convolutional neural networks are used so as to extract visual features. The latter are fused so as to form a single feature vector and are fed into a support vector machine for classification into human activities. For evaluation purposes, a challenging motion activity recognition dataset is used, while single-view, cross-view and cross-subject experiments are performed. Moreover, the proposed approach is compared to three other state-of-the-art methods, demonstrating superior performance in most experiments. <\/jats:p>","DOI":"10.1142\/s0129065723500028","type":"journal-article","created":{"date-parts":[[2022,12,27]],"date-time":"2022-12-27T14:40:26Z","timestamp":1672152026000},"source":"Crossref","is-referenced-by-count":13,"title":["A Multimodal Fusion Approach for Human Activity Recognition"],"prefix":"10.1142","volume":"33","author":[{"given":"Dimitrios","family":"Koutrintzes","sequence":"first","affiliation":[{"name":"Institute of Informatics and Telecommunications, National Center for Scientific Research \u2014 \u201cDemokritos\u201d, Athens, Greece"}]},{"given":"Evaggelos","family":"Spyrou","sequence":"additional","affiliation":[{"name":"Department of Informatics and Telecommunication, University of Thessaly, Lamia, Greece"}]},{"given":"Eirini","family":"Mathe","sequence":"additional","affiliation":[{"name":"Department of Informatics, Ionian University, Corfu, Greece"}]},{"given":"Phivos","family":"Mylonas","sequence":"additional","affiliation":[{"name":"Department of Informatics, Ionian University, Corfu, Greece"}]}],"member":"219","published-online":{"date-parts":[[2022,12,27]]},"reference":[{"key":"S0129065723500028BIB001","first-page":"265","volume-title":"12th USENIX Symp. Operating Systems Design and Implementation (OSDI 16)","author":"Abadi M.","year":"2016"},{"key":"S0129065723500028BIB002","doi-asserted-by":"publisher","DOI":"10.1007\/s00521-019-04359-7"},{"key":"S0129065723500028BIB003","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2019.2960588"},{"issue":"5","key":"S0129065723500028BIB004","doi-asserted-by":"crossref","first-page":"2250015","DOI":"10.1142\/S0129065722500150","volume":"32","author":"Avola D.","year":"2022","journal-title":"Int. J. Neural Syst."},{"key":"S0129065723500028BIB005","first-page":"3034","volume-title":"IEEE Conf. Computer Vision and Pattern Recognition","author":"Bilen H.","year":"2016"},{"key":"S0129065723500028BIB006","first-page":"1","volume-title":"IEEE Int. Conf. Advanced Video and Signal Based Surveillance (AVSS)","author":"Caetano C.","year":"2019"},{"key":"S0129065723500028BIB007","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.143"},{"key":"S0129065723500028BIB008","first-page":"91","volume-title":"IEEE Int. Conf. Computer Vision Workshops","author":"Chaaraoui A.","year":"2013"},{"key":"S0129065723500028BIB009","first-page":"1212","volume-title":"IEEE Int. Conf. Computer Vision","author":"Chen Y.","year":"2017"},{"issue":"3","key":"S0129065723500028BIB010","doi-asserted-by":"crossref","first-page":"1707","DOI":"10.1007\/s11042-019-08261-1","volume":"79","author":"Chen Y.","year":"2020","journal-title":"Multimedia Tools Appl."},{"issue":"2","key":"S0129065723500028BIB012","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1109\/MSP.2015.2503881","volume":"33","author":"Debes C.","year":"2016","journal-title":"IEEE Signal Process. Mag."},{"key":"S0129065723500028BIB013","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"S0129065723500028BIB014","first-page":"2969","volume-title":"Proc. IEEE\/CVF Conf. Computer Vision and Pattern Recognition","author":"Duan H.","year":"2022"},{"key":"S0129065723500028BIB015","doi-asserted-by":"crossref","first-page":"60736","DOI":"10.1109\/ACCESS.2019.2913393","volume":"7","author":"Ehatisham-Ul-Haq M.","year":"2019","journal-title":"IEEE Access"},{"key":"S0129065723500028BIB016","first-page":"5378","volume-title":"IEEE Conf. Computer Vision and Pattern Recognition","author":"Fernando B.","year":"2015"},{"key":"S0129065723500028BIB017","first-page":"601","volume-title":"27th ACM Int. Conf. Multimedia","author":"Gao X.","year":"2019"},{"key":"S0129065723500028BIB018","doi-asserted-by":"crossref","first-page":"473","DOI":"10.1145\/3453892.3461337","volume-title":"14th Pervasive Technologies Related to Assistive Environments Conf.","author":"Giannakos I.","year":"2021"},{"key":"S0129065723500028BIB019","volume-title":"Digital Image Processing","author":"Gonzalez R. C.","year":"2018"},{"key":"S0129065723500028BIB020","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2013.6638947"},{"key":"S0129065723500028BIB021","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2014.04.018"},{"issue":"1","key":"S0129065723500028BIB022","first-page":"762","volume-title":"AAAI Conf. Artif. Intell.","volume":"36","author":"Guo T.","year":"2022"},{"key":"S0129065723500028BIB023","first-page":"250","volume-title":"IEEE Int\u2019l Conf. Automatic Face & Gesture Recognition","author":"Haque M. A.","year":"2018"},{"key":"S0129065723500028BIB024","first-page":"242","volume-title":"Int. Conf. Machine Vision (ICMV)","volume":"10696","author":"Hatami N.","year":"2018"},{"key":"S0129065723500028BIB025","doi-asserted-by":"crossref","first-page":"e0228869","DOI":"10.1371\/journal.pone.0228869","volume":"15","author":"Hernandez V.","year":"2020","journal-title":"PLoS ONE"},{"issue":"3","key":"S0129065723500028BIB026","first-page":"807","volume":"28","author":"Hou Y.","year":"2016","journal-title":"IEEE Trans. CSVT"},{"key":"S0129065723500028BIB027","first-page":"5344","volume-title":"IEEE Conf. Computer Vision and Pattern Recognition","author":"Hu J. F.","year":"2015"},{"key":"S0129065723500028BIB028","doi-asserted-by":"crossref","first-page":"112","DOI":"10.1016\/j.ins.2019.10.047","volume":"513","author":"Huynh-The T.","year":"2020","journal-title":"Inf. Sci."},{"issue":"1","key":"S0129065723500028BIB029","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1007\/s12652-019-01239-9","volume":"11","author":"Imran J.","year":"2020","journal-title":"J. Ambient Intell. Humaniz. Comput."},{"key":"S0129065723500028BIB030","volume-title":"Fundamentals of Digital Image Processing","author":"Jain A. K.","year":"1989"},{"key":"S0129065723500028BIB031","first-page":"1307","volume-title":"23rd ACM Int. Conf. Multimedia","author":"Jiang W.","year":"2015"},{"key":"S0129065723500028BIB032","first-page":"1387","volume-title":"Int. Conf. Industrial Informatics (INDIN)","author":"Kalouris G.","year":"2019"},{"key":"S0129065723500028BIB033","first-page":"2025","volume-title":"IEEE Int. Conf. Image Processing (ICIP)","author":"Kao J. Y.","year":"2019"},{"key":"S0129065723500028BIB034","doi-asserted-by":"publisher","DOI":"10.1109\/LSP.2017.2690339"},{"key":"S0129065723500028BIB035","doi-asserted-by":"crossref","first-page":"959","DOI":"10.1109\/TIP.2019.2937757","volume":"29","author":"Ke Q.","year":"2019","journal-title":"IEEE Trans. Image Process."},{"issue":"4","key":"S0129065723500028BIB036","doi-asserted-by":"crossref","first-page":"e15704","DOI":"10.2196\/15704","volume":"8","author":"Keogh A.","year":"2020","journal-title":"JMIR mHealth uHealth"},{"issue":"2","key":"S0129065723500028BIB037","doi-asserted-by":"crossref","first-page":"648","DOI":"10.1109\/TPAMI.2021.3107160","volume":"44","author":"Koniusz P.","year":"2021","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"S0129065723500028BIB038","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1007\/978-3-031-08223-8_25","volume-title":"Int. Conf. Engineering Applications of Neural Networks","author":"Kostis I. A.","year":"2022"},{"key":"S0129065723500028BIB039","first-page":"370","volume-title":"Int. Conf. Pattern Recognition Applications and Methods \u2014 ICPRAM","author":"Koutrintzes D.","year":"2022"},{"key":"S0129065723500028BIB040","first-page":"2556","volume-title":"Int. Conf. Computer Vision","author":"Kuehne H.","year":"2011"},{"key":"S0129065723500028BIB041","first-page":"8172","volume-title":"IEEE\/CVF Conf. Computer Vision and Pattern Recognition","author":"Kwon T.","year":"2022"},{"key":"S0129065723500028BIB042","first-page":"1","volume-title":"IEEE Conf. Computer Vision and Pattern Recognition","author":"Laptev I.","year":"2008"},{"key":"S0129065723500028BIB043","doi-asserted-by":"publisher","DOI":"10.1109\/5.726791"},{"key":"S0129065723500028BIB044","first-page":"597","volume-title":"IEEE Int. Conf. Multimedia & Expo Workshops (ICMEW)","author":"Li C.","year":"2017"},{"key":"S0129065723500028BIB045","doi-asserted-by":"publisher","DOI":"10.1109\/LSP.2017.2678539"},{"key":"S0129065723500028BIB046","first-page":"585","volume-title":"IEEE Int. Conf. Multimedia & Expo Workshops","author":"Li C.","year":"2017"},{"key":"S0129065723500028BIB049","first-page":"872","volume-title":"Proc. IEEE\/CVF Int. Conf. Computer Vision","author":"Li T.","year":"2019"},{"key":"S0129065723500028BIB050","doi-asserted-by":"publisher","DOI":"10.3233\/ICA-200641"},{"key":"S0129065723500028BIB051","doi-asserted-by":"crossref","first-page":"23","DOI":"10.3233\/ICA-210661","volume":"29","author":"Liang Y.","year":"2022","journal-title":"Integr. Comput.-Aided Eng."},{"issue":"6","key":"S0129065723500028BIB053","doi-asserted-by":"crossref","first-page":"4788","DOI":"10.1109\/TIE.2018.2864702","volume":"66","author":"Liu C. L.","year":"2018","journal-title":"IEEE Trans. Ind. Electron."},{"key":"S0129065723500028BIB054","first-page":"1","volume-title":"Int\u2019l Conf. Digital Image Computing: Techniques and Applications","author":"Liu J.","year":"2017"},{"key":"S0129065723500028BIB055","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1016\/j.patcog.2017.02.030","volume":"68","author":"Liu M.","year":"2017","journal-title":"Pattern Recognit."},{"issue":"10","key":"S0129065723500028BIB056","doi-asserted-by":"crossref","first-page":"2684","DOI":"10.1109\/TPAMI.2019.2916873","volume":"42","author":"Liu J.","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"S0129065723500028BIB057","first-page":"10","volume-title":"CVPR Workshops","author":"Liu J.","year":"2019"},{"issue":"1","key":"S0129065723500028BIB058","doi-asserted-by":"crossref","first-page":"130","DOI":"10.3390\/s17010130","volume":"17","author":"Majumder S.","year":"2017","journal-title":"Sensors"},{"key":"S0129065723500028BIB059","doi-asserted-by":"publisher","DOI":"10.1140\/epjst\/e2008-00829-1"},{"key":"S0129065723500028BIB060","first-page":"6035","volume-title":"Int. Conf. Pattern Recognition (ICPR)","author":"Paoletti G.","year":"2021"},{"key":"S0129065723500028BIB061","series-title":"Communications in Computer and Information Science","doi-asserted-by":"crossref","first-page":"511","DOI":"10.1007\/978-3-030-20257-6_44","volume-title":"Engineering Applications of Neural Networks. EANN 2019","volume":"1000","author":"Papadakis A.","year":"2019"},{"key":"S0129065723500028BIB062","first-page":"258","volume-title":"IEEE Int. Symp. Image and Signal Processing and Analysis (ISPA)","author":"Papadakis A.","year":"2019"},{"key":"S0129065723500028BIB063","first-page":"1","volume-title":"ACM Int. Conf. Pervasive Technologies Related to Assistive Environments","author":"Papadakis A.","year":"2020"},{"key":"S0129065723500028BIB064","doi-asserted-by":"publisher","DOI":"10.1007\/s00521-019-04146-4"},{"issue":"8","key":"S0129065723500028BIB065","doi-asserted-by":"crossref","first-page":"1932","DOI":"10.3390\/s19081932","volume":"19","author":"Pham H. H.","year":"2019","journal-title":"Sensors"},{"key":"S0129065723500028BIB066","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2017.2682102"},{"issue":"8","key":"S0129065723500028BIB067","doi-asserted-by":"crossref","DOI":"10.1177\/1550147716665520","volume":"12","author":"Ranasinghe S.","year":"2016","journal-title":"Int. J. Distrib. Sens. Netw."},{"key":"S0129065723500028BIB068","first-page":"32","volume-title":"Int. Conf. Pattern Recognition (ICPR)","author":"Schuldt C.","year":"2004"},{"issue":"13","key":"S0129065723500028BIB069","doi-asserted-by":"crossref","first-page":"4342","DOI":"10.3390\/s21134342","volume":"21","author":"Silva V.","year":"2021","journal-title":"Sensors"},{"key":"S0129065723500028BIB071","first-page":"568","volume-title":"NIPS\u201914: Proceedings of the 27th International Conference on Neural Information Processing Systems","volume":"1","author":"Simonyan K.","year":"2014"},{"key":"S0129065723500028BIB072","doi-asserted-by":"publisher","DOI":"10.1023\/B:STCO.0000035301.49549.88"},{"issue":"4","key":"S0129065723500028BIB074","doi-asserted-by":"crossref","first-page":"55","DOI":"10.3390\/technologies8040055","volume":"8","author":"Spyrou E.","year":"2020","journal-title":"Technologies"},{"issue":"1","key":"S0129065723500028BIB075","first-page":"1929","volume":"15","author":"Srivastava N.","year":"2014","journal-title":"J. Mach. Learn. Res."},{"key":"S0129065723500028BIB076","doi-asserted-by":"crossref","first-page":"2892","DOI":"10.3390\/s18092892","volume":"18","author":"Steven Eyobu O.","year":"2018","journal-title":"Sensors"},{"key":"S0129065723500028BIB077","first-page":"121","volume-title":"Int. Conf. Engineering Applications of Neural Networks","author":"Stylianou-Nikolaidou S.","year":"2021"},{"key":"S0129065723500028BIB078","first-page":"2147","volume-title":"Int. Conf. Computer Vision","author":"Sun L.","year":"2017"},{"key":"S0129065723500028BIB080","first-page":"3104","volume-title":"NIPS\u201914: Proceedings of the 27th International Conference on Neural Information Processing Systems","volume":"2","author":"Sutskever I.","year":"2014"},{"key":"S0129065723500028BIB081","first-page":"5323","volume-title":"IEEE Conf. Computer Vision and Pattern Recognition","author":"Tang Y.","year":"2018"},{"issue":"6","key":"S0129065723500028BIB082","doi-asserted-by":"crossref","first-page":"2675","DOI":"10.3390\/app11062675","volume":"11","author":"Tasnim N.","year":"2021","journal-title":"Appl. Sci."},{"key":"S0129065723500028BIB083","doi-asserted-by":"crossref","DOI":"10.1201\/b10676","volume-title":"Graphics and Visualization: Principles & Algorithms","author":"Theoharis T.","year":"2008"},{"issue":"6","key":"S0129065723500028BIB084","doi-asserted-by":"crossref","first-page":"671","DOI":"10.1007\/s00530-020-00677-2","volume":"26","author":"Verma P.","year":"2020","journal-title":"Multimedia Syst."},{"key":"S0129065723500028BIB085","doi-asserted-by":"crossref","first-page":"325","DOI":"10.1145\/3316782.3322740","volume-title":"ACM Int. Conf. PErvasive Technologies Related to Assistive Environments","author":"Vernikos I.","year":"2019"},{"key":"S0129065723500028BIB086","first-page":"1","volume-title":"2019 14th International Workshop on Semantic and Social Media Adaptation and Personalization (SMAP)","author":"Vernikos I.","year":"2019"},{"issue":"28","key":"S0129065723500028BIB087","volume":"2","author":"Vrigkas M.","year":"2015","journal-title":"Front. Robot. AI"},{"key":"S0129065723500028BIB088","first-page":"102","volume-title":"ACM Int. Conf. Multimedia","author":"Wang P.","year":"2016"},{"key":"S0129065723500028BIB089","doi-asserted-by":"publisher","DOI":"10.1016\/j.cviu.2018.04.007"},{"key":"S0129065723500028BIB090","first-page":"20","volume-title":"2012 IEEE Computer Society Conf. Computer Vision and Pattern Recognition Workshops","author":"Xia L.","year":"2012"},{"issue":"8","key":"S0129065723500028BIB091","first-page":"2405","volume":"29","author":"Yang Z.","year":"2018","journal-title":"IEEE Trans. CSVT"},{"key":"S0129065723500028BIB092","first-page":"2117","volume-title":"IEEE Int. Conf. Computer Vision","author":"Zhang P.","year":"2017"},{"key":"S0129065723500028BIB093","doi-asserted-by":"crossref","first-page":"1061","DOI":"10.1109\/TIP.2019.2937724","volume":"29","author":"Zhang P.","year":"2019","journal-title":"IEEE Trans. Image Process."},{"key":"S0129065723500028BIB094","first-page":"1112","volume-title":"IEEE\/CVF Conf. Computer Vision and Pattern Recognition","author":"Zhang P.","year":"2020"},{"issue":"9","key":"S0129065723500028BIB095","doi-asserted-by":"crossref","first-page":"2330","DOI":"10.1109\/TMM.2018.2802648","volume":"2","author":"Zhang S.","year":"2018","journal-title":"IEEE Trans. Multimedia"},{"key":"S0129065723500028BIB096","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2017.2684186"}],"container-title":["International Journal of Neural Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.worldscientific.com\/doi\/pdf\/10.1142\/S0129065723500028","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,16]],"date-time":"2023-01-16T05:50:57Z","timestamp":1673848257000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.worldscientific.com\/doi\/10.1142\/S0129065723500028"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,27]]},"references-count":89,"journal-issue":{"issue":"01","published-print":{"date-parts":[[2023,1]]}},"alternative-id":["10.1142\/S0129065723500028"],"URL":"https:\/\/doi.org\/10.1142\/s0129065723500028","relation":{},"ISSN":["0129-0657","1793-6462"],"issn-type":[{"value":"0129-0657","type":"print"},{"value":"1793-6462","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,27]]},"article-number":"2350002"}}