{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,8]],"date-time":"2025-11-08T17:54:52Z","timestamp":1762624492445,"version":"build-2065373602"},"reference-count":24,"publisher":"MDPI AG","issue":"20","license":[{"start":{"date-parts":[[2020,10,17]],"date-time":"2020-10-17T00:00:00Z","timestamp":1602892800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61205075","61471260"],"award-info":[{"award-number":["61205075","61471260"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Low frequency vibration monitoring has significant implications on environmental safety and engineering practices. Vibration expressed by visual information should contain sufficient spatial information. RGB-D camera could record diverse spatial information of vibration in frame images. Deep learning can adaptively transform frame images into deep abstract features through nonlinear mapping, which is an effective method to improve the intelligence of vibration monitoring. In this paper, a multi-modal low frequency visual vibration monitoring system based on Kinect v2 and 3DCNN-ConvLSTM is proposed. Microsoft Kinect v2 collects RGB and depth video information of vibrating objects in unstable ambient light. The 3DCNN-ConvLSTM architecture can effectively learn the spatial-temporal characteristics of muti-frequency vibration. The short-term spatiotemporal feature of the collected vibration information is learned through 3D convolution networks and the long-term spatiotemporal feature is learned through convolutional LSTM. Multi-modal fusion of RGB and depth mode is used to further improve the monitoring accuracy to 93% in the low frequency vibration range of 0\u201310 Hz. The results show that the system can monitor low frequency vibration and meet the basic measurement requirements.<\/jats:p>","DOI":"10.3390\/s20205872","type":"journal-article","created":{"date-parts":[[2020,10,17]],"date-time":"2020-10-17T05:45:51Z","timestamp":1602913551000},"page":"5872","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Low Frequency Vibration Visual Monitoring System Based on Multi-Modal 3DCNN-ConvLSTM"],"prefix":"10.3390","volume":"20","author":[{"given":"Alimina","family":"Alimasi","sequence":"first","affiliation":[{"name":"School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China"}]},{"given":"Hongchen","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China"}]},{"given":"Chengang","family":"Lyu","sequence":"additional","affiliation":[{"name":"School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China"}]}],"member":"1968","published-online":{"date-parts":[[2020,10,17]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"150","DOI":"10.1016\/j.apacoust.2019.05.011","article-title":"Contribution of large-vehicle vibration and bridge vibration to low-frequency noise generation characteristics","volume":"155","author":"Iwabuki","year":"2019","journal-title":"Appl. Acoust."},{"key":"ref_2","first-page":"271","article-title":"Impact of Traffic-Induced Vibrations on Residential Buildings and Their Occupants in Metropolitan Cities","volume":"31","author":"Erkal","year":"2019","journal-title":"Promet Traffic Transp."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1016\/j.apergo.2015.11.010","article-title":"Discomfort of seated persons exposed to low frequency lateral and roll oscillation: Effect of backrest height","volume":"54","author":"Beard","year":"2016","journal-title":"Appl. Ergon."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"255","DOI":"10.1007\/s11340-013-9784-8","article-title":"Vibration Monitoring of Multiple Bridge Points by Means of a Unique Vision-Based Measuring System","volume":"54","author":"Busca","year":"2014","journal-title":"Exp. Mech."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1016\/j.autcon.2018.05.025","article-title":"Multi-point vibration measurement and mode magnification of civil structures using video-based motion processing","volume":"93","author":"Shang","year":"2018","journal-title":"Autom. Constr."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Zhang, J., Yang, Z., and Deng, H. (2019). Dynamic Visual Measurement of Driver Eye Movements. Sensors, 19.","DOI":"10.3390\/s19102217"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Scovanner, P., Ali, S., and Shah, M. (2007, January 24\u201329). A 3-Dimensional SIFT Descriptor and its Application to Action Recognition. Proceedings of the ACM International Multimedia Conference and Exhibition, Augsburg, Germany.","DOI":"10.1145\/1291233.1291311"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Liu, J., and Yang, X. (2018). Learning to See the Vibration: A Neural Network for Vibration Frequency Prediction. Sensors, 18.","DOI":"10.20944\/preprints201807.0086.v1"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"107113","DOI":"10.1016\/j.measurement.2019.107113","article-title":"Non-contact low-frequency vibration Rapid Measurement Based on hue-height mapping","volume":"151","author":"Lyu","year":"2019","journal-title":"Measurement"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"869","DOI":"10.1016\/j.jsv.2004.10.043","article-title":"Structural damage detection using digital video imaging technique and wavelet transformation","volume":"286","author":"Poudel","year":"2005","journal-title":"J. Sound Vib."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1192","DOI":"10.1109\/SURV.2012.110112.00192","article-title":"A Survey on Human Activity Recognition using Wearable Sensors","volume":"15","author":"Lara","year":"2013","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"3246","DOI":"10.1109\/TGRS.2019.2951445","article-title":"Heterogeneous Transfer Learning for Hyperspectral Image Classification Based on Convolutional Neural Network","volume":"58","author":"He","year":"2020","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Ha, S., Yun, J.-M., and Choi, S. (2015, January 9\u201312). Multi-Modal Convolutional Neural Networks for Activity Recognition. Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC), Hong Kong.","DOI":"10.1109\/SMC.2015.525"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"106885","DOI":"10.1016\/j.ymssp.2020.106885","article-title":"CNN-LSTM deep learning architecture for computer vision-based modal frequency detection","volume":"144","author":"Yang","year":"2020","journal-title":"Mech. Syst. Signal Process."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1016\/j.jsv.2019.05.027","article-title":"VibroNet: Recurrent neural networks with multi-target learning for image-based vibration frequency measurement","volume":"457","author":"Liu","year":"2019","journal-title":"J. Sound Vib."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"268","DOI":"10.1016\/j.measurement.2019.04.093","article-title":"A deep convolutional neural network based fusion method of two-direction vibration signal data for health state identification of planetary gearboxes","volume":"146","author":"Chen","year":"2019","journal-title":"Measurement"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"182","DOI":"10.1016\/j.compind.2018.12.013","article-title":"A novel convolutional neural network based fault recognition method via image fusion of multi-vibration-signals","volume":"105","author":"Wang","year":"2019","journal-title":"Comput. Ind."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1016\/j.inffus.2018.06.002","article-title":"Data fusion and multiple classifier systems for human activity detection and health monitoring: Review and open research directions","volume":"46","author":"Nweke","year":"2019","journal-title":"Inf. Fusion"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1001","DOI":"10.1002\/hbm.24428","article-title":"Effective feature learning and fusion of multimodality data using stage-wise deep neural network for dementia diagnosis","volume":"40","author":"Zhou","year":"2019","journal-title":"Hum. Brain Mapp."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"5558","DOI":"10.1109\/LRA.2020.3007457","article-title":"Real-Time Fusion Network for RGB-D Semantic Segmentation Incorporating Unexpected Obstacle Detection for Road-Driving Images","volume":"5","author":"Sun","year":"2020","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Liu, W., Li, F., and Jing, C. (2020). Recognition and location of typical automotive parts based on the RGB-D camera. Complex Intell. Syst., 1\u20137.","DOI":"10.1007\/s40747-020-00182-z"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1109\/TPAMI.2012.59","article-title":"3D Convolutional Neural Networks for Human Action Recognition","volume":"35","author":"Ji","year":"2013","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015, January 7\u201313). Learning Spatiotemporal Features with 3D Convolutional Networks. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.510"},{"key":"ref_24","unstructured":"Shi, X., Chen, Z., and Wang, H. (2015). Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting. Advances in Neural Information Processing Systems, Nips."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/20\/5872\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:23:04Z","timestamp":1760178184000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/20\/5872"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,10,17]]},"references-count":24,"journal-issue":{"issue":"20","published-online":{"date-parts":[[2020,10]]}},"alternative-id":["s20205872"],"URL":"https:\/\/doi.org\/10.3390\/s20205872","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2020,10,17]]}}}